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The most significant differences between this edition and the first are as follows: 

• Additional chapters and sections have been written, dealing with: 

— nonlinear controllability via Lie-algebraic methods, 

— variational and numerical approaches to nonlinear control, including 
a brief introduction to the Calculus of Variations and the Minimum 
Principle, 

— time-optimal control of linear systems, 

— feedback linearization (single- input case), 

— nonlinear optimal feedback, 

— controllability of recurrent nets, and 

— controllability of linear systems with bounded controls. 

• The discussion on nonlinear stabilization has been expanded, introducing 
the basic ideas of control-Lyapunov functions, backstepping, and damping 
control. 

• The chapter on dynamic programming and linear-quadratic problems has 
been substantially edited, so that the material on linear systems can be 
read in a fully independent manner from the nonlinear preliminaries. 

• A fairly large number of errors and typos have been corrected. 

• A list of symbols has been added. 

I would like to strongly encourage readers to send me suggestions and com- 
ments by e-mail (sontag@control.rutgers.edu) , and also to visit the following 
Web site: 



http://www.math.rutgers.edu/ sontag/ 
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where I expect to post updates, additional material and references, links, and 
errata. 

The current contents of the text far exceed what can be done in a year, 
if all material is covered and complete proofs are given in lectures. However, 
there are several ways to structure a year-long course, or two such courses, 
based on parts of the book. For example, one may cover only the linear theory, 
skipping the optional sections as well as the chapters on nonlinear controllability 
and multiplier (variational) methods. A separate course, fairly independent, 
could cover the more advanced nonlinear material. Ultimately, the topics should 
reflect student and faculty background and interests, and I'll be happy to discuss 
syllabi with potential instructors. 

I wish to thank all those colleagues, students, and readers who have sent 
me suggestions and comments, including in particular Brian Ingalls, Gerardo 
Lafferriere, Michael Malisoff, and Konrad Reif. A most special acknowledgment 
goes to Jose Luis Mantilla Aguilar and Sarah Koskie, who pointed out a very 
large number of typos and errors, and proposed appropriate corrections. Of 
course, many mistakes surely remain, and I am solely responsible for them. I 
also reiterate my acknowledgment and thanks for the continued support from 
the Air Force Office of Scientific Research, and to my family for their infinite 
patience. 



Eduardo D. Sontag 



Piscataway, NJ 
May, 1998. 



Preface to the First Edition 



This textbook introduces the basic concepts and results of mathematical control 
and system theory. Based on courses that I have taught during the last 15 
years, it presents its subject in a self-contained and elementary fashion. It is 
geared primarily to an audience consisting of mathematically mature advanced 
undergraduate or beginning graduate students. In addition, it can be used by 
engineering students interested in a rigorous, proof-oriented systems course that 
goes beyond the classical frequency-domain material and more applied courses. 

The minimal mathematical background that is required of the reader is a 
working knowledge of linear algebra and differential equations. Elements of 
the theory of functions of a real variable, as well as elementary notions from 
other areas of mathematics, are used at various points and are reviewed in the 
appendixes. However, the book was written in such a manner that readers not 
comfortable with these techniques should still be able to understand all concepts 
and to follow most arguments, at the cost of skipping a few technical details 
and making some simplifying assumptions in a few places — such as dealing only 
with piecewise continuous functions where arbitrary measurable functions are 
allowed. In this dual mode, I have used the book in courses at Rutgers University 
with mixed audiences consisting of mathematics, computer science, electrical 
engineering, and mechanical and aerospace engineering students. Depending 
on the detail covered in class, it can be used for a one-, two-, or three-semester 
course. By omitting the chapter on optimal control and the proofs of the results 
in the appendixes, a one-year course can be structured with no difficulty 

The book covers what constitutes the common core of control theory: The al- 
gebraic theory of linear systems, including controllability, observability, feedback 
equivalence, and minimality; stability via Lyapunov, as well as input/output 
methods; ideas of optimal control; observers and dynamic feedback; parameter- 
ization of stabilizing controllers (in the scalar case only); and some very basic 
facts about frequency domain such as the Nyquist criterion. Kalman filtering 
is introduced briefly through a deterministic version of "optimal observation;" 
this avoids having to develop the theory of stochastic processes and represents 
a natural application of optimal control and observer techniques. In general, no 
stochastic or infinite dimensional results are covered, nor is a detailed treatment 
given of nonlinear differential-geometric control; for these more advanced areas, 
there are many suitable texts. 
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The introductory chapter describes the main contents of the book in an in- 
tuitive and informal manner, and the reader would be well advised to read this 
in detail. I suggest spending at least a week of the course covering this mate- 
rial. Sections marked with asterisks can be left out without loss of continuity. I 
have only omitted the proofs of what are labeled "Lemma/Exercises," with the 
intention that these proofs must be worked out by the reader in detail. Typi- 
cally, Lemma/Exercises ask the reader to prove that some elementary property 
holds, or to prove a difference equation analogue of a differential equation re- 
sult just shown. Only a trivial to moderate effort is required, but filling in the 
details forces one to read the definitions and proofs carefully. "Exercises" are 
for the most part also quite simple; those that are harder are marked with the 
symbol O . 

Control and system theory shares with some other areas of "modern" ap- 
plied mathematics (such as quantum field theory, geometric mechanics, and 
computational complexity), the characteristic of employing a broad range of 
mathematical tools, providing many challenges and possibilities for interactions 
with established areas of "pure" mathematics. Papers often use techniques 
from algebraic and differential geometry, functional analysis, Lie algebras, com- 
binatorics, and other areas, in addition to tools from complex variables, linear 
algebra, and ordinary and partial differential equations. While staying within 
the bounds of an introductory presentation, I have tried to provide pointers 
toward some of these exciting directions of research, particularly through the 
remarks at the ends of chapters and through references to the literature. (At 
a couple of points I include further details, such as when I discuss Lie group 
actions and families of systems, or degree theory and nonlinear stabilization, 
or ideal theory and finite-experiment observability, but these are restricted to 
small sections that can be skipped with no loss of continuity.) 

This book should be useful too as a research reference source, since I have 
included complete proofs of various technical results often used in papers but 
for which precise citations are hard to find. I have also compiled a rather long 
bibliography covering extensions of the results given and other areas not treated, 
as well as a detailed index facilitating access to these. 

Although there are hundreds of books dealing with various aspects of control 
and system theory, including several extremely good ones, this text is unique in 
its emphasis on foundational aspects. I know of no other book that, while cov- 
ering the range of topics treated here and written in a standard theorem/proof 
style, also develops the necessary techniques from scratch and does not assume 
any background other than basic mathematics. In fact, much effort was spent 
trying to find consistent notations and terminology across these various areas. 
On the other hand, this book does not provide realistic engineering examples, as 
there are already many books that do this well. (The bibliography is preceded 
by a discussion of such other texts.) 

I made an effort to highlight the distinctions as well as the similarities be- 
tween continuous- and discrete-time systems, and the process (sampling) that 
is used in practice for computer control. In this connection, I find it highly 
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probable that future developments in control theory will continue the move- 
ment toward a more "computer-oriented" and "digital/logical" view of systems, 
as opposed to the more classical continuous-time smooth paradigm motivated 
by analogue devices. Because of this, it is imperative that a certain minimal 
amount of the 'abstract nonsense' of systems (abstract definitions of systems as 
actions over general time sets, and so forth) be covered: It is impossible to pose, 
much less solve, systems design problems unless one first understands what a 
system is. This is analogous to understanding the definition of function as a set 
of ordered pairs, instead of just thinking of functions as only those that can be 
expressed by explicit formulas. 

A few words about notation and numbering conventions. Except for theo- 
rems, which are numbered consecutively, all other environments (lemmas, def- 
initions, and so forth) are numbered by section, while equations are numbered 
by chapter. In formal statements, I use roman characters such as x to denote 
states at a given instant, reserving Greek letters such as £ for state trajectories, 
and a similar convention is used for control values versus control functions, and 
output values versus output functions; however, in informal discussions and ex- 
amples, I use roman notation for both points and functions, the meaning being 
clear from the context. The symbol I marks an end of proof, while □ indicates 
the end of a remark, example, etc., as well as the end of a statement whose 
proof has been given earlier. 

This volume represents one attempt to address two concerns raised in the 
report "Future Directions in Control Theory: A Mathematical Perspective," 
which was prepared by an international panel under the auspices of various 
American funding agencies (National Science Foundation, Air Force Office of 
Scientific Research, Army Research Office, and Office of Naval Research) and 
was published in 1988 by the Society for Industrial and Applied Mathematics. 
Two of the main recommendations of the report were that further efforts be 
made to achieve conceptual unity in the field, as well as to develop better train- 
ing for students and faculty in mathematics departments interested in being 
exposed to the area. Hopefully, this book will be useful in helping achieve these 
goals. 

It is hard to even begin to acknowledge all those who influenced me pro- 
fessionally and from whom I learned so much, starting with Enzo Gentile and 
other faculty at the Mathematics Department of the University of Buenos Aires. 
Without doubt, my years at the Center for Mathematical System Theory as a 
student of Rudolf Kalman were the central part of my education, and the stim- 
ulating environment of the center was unequaled in its possibilities for learn- 
ing. To Professor Kalman and his long-range view of the area, originality of 
thought, and emphasis on critical thinking, I will always owe major gratitude. 
Others who spent considerable time at the Center, and from whose knowledge 
I also benefited immensely at that time and ever since, include Roger Brockett, 
Samuel Eilenberg, Michel Fliess, Malo Hautus, Michiel Hazewinkel, Hank Her- 
mes, Michel Heymann, Bruce Lee, Alberto Isidori, Ed Kamen, Sanjoy Mitter, 
Yves Rouchaleau, Yutaka Yamamoto, Jan Willems, and Bostwick Wyman. In 
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latter years I learned much from many people, but I am especially indebted to 
my Rutgers colleague Hector Sussmann, who introduced me to the continuous- 
time nonlinear theory. While writing this book, I received constant and very 
useful feedback from graduate students at Rutgers, including Francesca Alber- 
tini, Yuandan Lin, Wen-Sheng Liu, Guoqing Tang, Yuan Wang, and Yudi Yang. 
I wish to especially thank Renee Schwarzschild, who continuously provided me 
with extensive lists of misprints, errors, and comments. Both Pramod Khar- 
gonekar and Jack Rugh also gave me useful suggestions at various points. 

The continued and generous research support that the Air Force Office of 
Scientific Research provided to me during most of my career was instrumental in 
my having the possibility of really understanding much of the material treated 
in this book as well as related areas. I wish to express my sincere gratitude for 
AFOSR support of such basic research. 

Finally, my special thanks go to my wife Fran and to my children Laura 
and David, for their patience and understanding during the long time that this 
project took. 



Eduardo D. Sontag 



Piscataway, NJ 
June, 1990. 
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Chapter 1 

Introduction 

1.1 What Is Mathematical Control Theory? 

Mathematical control theory is the area of application-oriented mathematics 
that deals with the basic principles underlying the analysis and design of control 
systems. To control an object means to influence its behavior so as to achieve a 
desired goal. In order to implement this influence, engineers build devices that 
incorporate various mathematical techniques. These devices range from Watt's 
steam engine governor, designed during the English Industrial Revolution, to 
the sophisticated microprocessor controllers found in consumer items — such as 
CD players and automobiles — or in industrial robots and airplane autopilots. 

The study of these devices and their interaction with the object being con- 
trolled is the subject of this book. While on the one hand one wants to un- 
derstand the fundamental limitations that mathematics imposes on what is 
achievable, irrespective of the precise technology being used, it is also true that 
technology may well influence the type of question to be asked and the choice 
of mathematical model. An example of this is the use of difference rather than 
differential equations when one is interested in digital control. 

Roughly speaking, there have been two main lines of work in control the- 
ory, which sometimes have seemed to proceed in very different directions but 
which are in fact complementary. One of these is based on the idea that a good 
model of the object to be controlled is available and that one wants to some- 
how optimize its behavior. For instance, physical principles and engineering 
specifications can be — and are — used in order to calculate that trajectory of 
a spacecraft which minimizes total travel time or fuel consumption. The tech- 
niques here are closely related to the classical calculus of variations and to other 
areas of optimization theory; the end result is typically a preprogrammed flight 
plan. The other main line of work is that based on the constraints imposed 
by uncertainty about the model or about the environment in which the object 
operates. The central tool here is the use of feedback in order to correct for 
deviations from the desired behavior. For instance, various feedback control 
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systems are used during actual space flight in order to compensate for errors 
from the precomputed trajectory. Mathematically stability theory dynamical 
systems, and especially the theory of functions of a complex variable, have had 
a strong influence on this approach. It is widely recognized today that these 
two broad lines of work deal just with different aspects of the same problems, 
and we do not make an artificial distinction between them in this book. 

Later on we shall give an axiomatic definition of what we mean by a "system" 
or "machine." Its role will be somewhat analogous to that played in mathematics 
by the definition of "function" as a set of ordered pairs: not itself the object of 
study, but a necessary foundation upon which the entire theoretical development 
will rest. In this Chapter, however, we dispense with precise definitions and will 
use a very simple physical example in order to give an intuitive presentation of 
some of the goals, terminology, and methodology of control theory. 

The discussion here will be informal and not rigorous, but the reader is 
encouraged to follow it in detail, since the ideas to be given underlie everything 
else in the book. Without them, many problems may look artificial. Later, we 
often refer back to this Chapter for motivation. 



1.2 Proportional-Derivative Control 

One of the simplest problems in robotics is that of controlling the position of a 
single-link rotational joint using a motor placed at the pivot. Mathematically, 
this is just a pendulum to which one can apply a torque as an external force 
(see Figure 1.1). 




Figure 1.1: Pendulum. 

We assume that friction is negligible, that all of the mass is concentrated 
at the end, and that the rod has unit length. From Newton's law for rotating 
objects, there results, in terms of the variable 6 that describes the counterclock- 
wise angle with respect to the vertical, the second-order nonlinear differential 
equation 

m9(t)+mg sin 6(t) = u(t), (1.1) 
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where m is the mass, g the acceleration due to gravity, and u(t) the value of 
the external torque at time t (counterclockwise being positive). We call u(-) 
the input or control function. To avoid having to keep track of constants, let us 
assume that units of time and distance have been chosen so that m = g = 1 . 

The vertical stationary position (9 = ir,9 = 0) is an equilibrium when no 
control is being applied (u = 0), but a small deviation from this will result in 
an unstable motion. Let us assume that our objective is to apply torques as 
needed to correct for such deviations. For small 9 — n. 



Here we use the standard "little-o" notation: o{x) stands for some function g(x) 

for which 

q(x) 
lim ^-^- = . 

x^O X 

Since only small deviations are of interest, we drop the nonlinear part repre- 
sented by the term o{9 — ir). Thus, with ip := 9 — n as a new variable, we replace 
equation (1.1) by the linear differential equation 

<p(t) - <p(t) = u(t) (1.2) 

as our object of study. (See Figure 1.2.) Later we will analyze the effect of the 
ignored nonlinearity. 




Figure 1.2: Inverted pendulum. 

Our objective then is to bring ip and (p to zero, for any small nonzero initial 
values </?(0),<p(0) in equation (1.2), and preferably to do so as fast as possible, 
with few oscillations, and without ever letting the angle and velocity become 
too large. Although this is a highly simplified system, this kind of "servo" 
problem illustrates what is done in engineering practice. One typically wants 
to achieve a desired value for certain variables, such as the correct idling speed 
in an automobile's electronic ignition system or the position of the read/write 
head in a disk drive controller. 

A naive first attempt at solving this control problem would be as follows: If 
we are to the left of the vertical, that is, if tp = 9 — -k > 0, then we wish to move 
to the right, and therefore, we apply a negative torque. If instead we are to 
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the right, we apply a positive, that is to say counterclockwise, torque. In other 
words, we apply proportional feedback 

u(t) = -aip(t) , (1.3) 

where a is some positive real number, the feedback gain. 

Let us analyze the resulting closed-loop equation obtained when the value 
of the control given by (1.3) is substituted into the open-loop original equation 
(1.2), that is 

<p(t) - ip(t) + aip(t) = . (1.4) 

If a > 1 , the solutions of this differential equation are all oscillatory, since the 
roots of the associated characteristic equation 

z 2 + a -l = (1.5) 

are purely imaginary, z = ±iy/a — 1. If instead a < 1, then all of the solutions 
except for those with 

ip(0) = -(p(0)y/l -a 

diverge to ±oo. Finally, if a = 1, then each set of initial values with </?(0) = is 
an equilibrium point of the closed-loop system. Therefore, in none of the cases 
is the system guaranteed to approach the desired configuration. 

We have seen that proportional control does not work. We proved this for 
the linearized model, and an exercise below will show it directly for the original 
nonlinear equation (1.1). Intuitively, the problem can be understood as follows. 
Take first the case a < 1. For any initial condition for which y(0) is small 
but positive and <p(0) = 0, there results from equation (1.4) that </3(0) > 0. 
Therefore, also (f> and hence (p increase, and the pendulum moves away, rather 
than toward, the vertical position. When a > 1 the problem is more subtle: 
The torque is being applied in the correct direction to counteract the natural 
instability of the pendulum, but this feedback helps build too much inertia. 
In particular, when already close to </?(0) = but moving at a relatively large 
speed, the controller (1.3) keeps pushing toward the vertical, and overshoot and 
eventual oscillation result. 

The obvious solution is to keep a > 1 but to modify the proportional feed- 
back (1.3) through the addition of a term that acts as a brake, penalizing ve- 
locities. In other words, one needs to add damping to the system. We arrive 
then at a PD, or proportional-derivative feedback law, 

u(t) = -aip{t) - P<p{t) , (1.6) 

with a > 1 and f3 > 0. In practice, implementing such a controller involves 
measurement of both the angular position and the velocity. If only the former 
is easily available, then one must estimate the velocity as part of the control 
algorithm; this will lead later to the idea of observers, which are techniques for 
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reliably performing such an estimation. We assume here that <p can indeed be 
measured. Consider then the resulting closed-loop system, 

0(t) +/?£(*) + (a- 1M*)=0. (1-7) 

The roots of its associated characteristic equation 

z 2 + f3z + a-l = (1.8) 

are 

-/3±V/3 2 -4(q-l) 

2 
both of which have negative real parts. Thus all the solutions of (1.2) converge 
to zero. The system has been stabilized under feedback. This convergence may 
be oscillatory, but if we design the controller in such a way that in addition to 
the above conditions on a and j3 it is true that 

/3 2 >4(a-l), (1.9) 

then all of the solutions are combinations of decaying exponentials and no os- 
cillation results. 

We conclude from the above discussion that through a suitable choice of 
the gains a and (3 it is possible to attain the desired behavior, at least for the 
linearized model. That this same design will still work for the original nonlinear 
model, and, hence, assuming that this model was accurate, for a real pendulum, 
is due to what is perhaps the most important fact in control theory — and for 
that matter in much of mathematics — namely that first-order approximations 
are sufficient to characterize local behavior. Informally, we have the following 
linearization principle: 



Designs based on linearizations work locally for the original system 



The term "local" refers to the fact that satisfactory behavior only can be ex- 
pected for those initial conditions that are close to the point about which the 
linearization was made. Of course, as with any "principle," this is not a theorem. 
It can only become so when precise meanings are assigned to the various terms 
and proper technical assumptions are made. Indeed, we will invest some effort 
in this text to isolate cases where this principle may be made rigorous. One of 
these cases will be that of stabilization, and the theorem there will imply that 
if we can stabilize the linearized system (1.2) for a certain choice of parameters 
a, (3 in the law (1.6), then the same control law does bring initial conditions of 
(1.1) that start close to 6 = it, 6 = to the vertical equilibrium. 

Basically because of the linearization principle, a great deal of the literature 
in control theory deals exclusively with linear systems. From an engineering 
point of view, local solutions to control problems are often enough; when they 
are not, ad hoc methods sometimes may be used in order to "patch" together 
such local solutions, a procedure called gain scheduling. Sometimes, one may 
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even be lucky and find a way to transform the problem of interest into one that 
is globally linear; we explain this later using again the pendulum as an example. 
In many other cases, however, a genuinely nonlinear approach is needed, and 
much research effort during the past few years has been directed toward that 
goal. In this text, when we develop the basic definitions and results for the 
linear theory we will always do so with an eye toward extensions to nonlinear, 
global, results. 

An Exercise 

As remarked earlier, proportional control (1.3) by itself is inadequate for the 
original nonlinear model. Using again <p = 9 — 71", the closed-loop equation 
becomes 

(p(t) - sin ip{t) + aip{t) = 0. (1.10) 

The next exercise claims that solutions of this equation typically will not ap- 
proach zero, no matter how the feedback gain a is picked. 

Exercise 1.2.1 Assume that a is any fixed real number, and consider the ("en- 
ergy" ) function of two real variables 

V{x,y) := cosx- 1 + -(ax 2 + y 2 ) . (1-H) 

Show that V(ip(t),tp(t)) is constant along the solutions of (1.10). Using that 
U(x,0) is an analytic function and therefore that its zero at x = is isolated, 
conclude that there are initial conditions of the type (p(0) = £, <^(0) = 0, with e 
arbitrarily small, for which the corresponding solution of (1-10) does not satisfy 
that ip(t) — > and ip(t) — > as t —> oo. □ 



1.3 Digital Control 

The actual physical implementation of (1.6) need not concern us here, but some 
remarks are in order. Assuming again that the values ip(t) and (f(t), or equiva- 
lently 6(t) and 0(t), can be measured, it is necessary to take a linear combina- 
tion of these in order to determine the torque u(t) that the motor must apply. 
Such combinations are readily carried out by circuits built out of devices called 
operational amplifiers. Alternatively, the damping term can be separately im- 
plemented directly through the use of an appropriate device (a "dashpot"), and 
the torque is then made proportional to f(i). 

A more modern alternative, attractive especially for larger systems, is to 
convert position and velocity to digital form and to use a computer to calcu- 
late the necessary controls. Still using the linearized inverted pendulum as an 
illustration, we now describe some of the mathematical problems that this leads 
to. 
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A typical approach to computer control is based on the sample- and-hold 
technique, which can be described as follows. The values ip(t) and <p(t) are 
measured only at discrete instants or sampling times 



0, 8, 25, 3(5, 



, k8, 



The control law is updated by a program at each time t = k5 on the basis of 
the sampled values (p(k5) and ip(k6). The output of this program, a value Vk, 
is then fed into the system as a control (held constant at that value) during the 
interval [kS, kd + 8] . 
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Figure 1.3: Sampled control. 

For simplicity we assume here that the computation of v k can be done quickly 
relative to the length 5 of the sampling intervals; otherwise, the model must be 
modified to account for the extra delay. To calculate the effect of applying the 
constant control 

u{t) = v k if t e [k8,kS + 8] (1.12) 

we solve the differential equation (1.2) with this function u. By differentiation 
one can verify that the general solution is, for t G [kd, kS + S\, 



n( .-, JfjkS) + ipjkS) + v k t _ kS ip(kS) - ip(k5) + v k t+kS 
<p(t) = e H e T —Vk, (1-13) 



si » 



• / f) = f(k8) + y{k6) + v k ct _ kS _ ipjkS) - ipjkS) + v k c _ t+kS 



Thus, applying the constant control u gives rise to new values for ip(k5 + 5) and 
<p(kd + 5) at the end of the interval via the formula 



($l8H($§)+»». 



(1.15) 
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where 

a _ /cosh (5 sinh<5\ , , 

y sinh 5 cosh S J 

and 

In other words, if we let Xq,X\,... denote the sequence of two dimensional 
vectors 

'<p(kS)\ 



x k := 



ip{kS) J ' 



then {x k } satisfies the recursion 

x k+1 = Ax k + Bv k . (1.18) 

Assume now that we wish to program our computer to calculate the constant 
control values v k to be applied during any interval via a linear transformation 

v k :=Fx k (1.19) 

of the measured values of position and velocity at the start of the interval. Here 
F is just a row vector (/i, f 2 ) that gives the coefficients of a linear combination 
of these measured values. Formally we are in a situation analogous to the PD 
control (1.6), except that we now assume that the measurements are being 
made only at discrete times and that a constant control will be applied on each 
interval. Substituting (1.19) into the difference equation (1.18), there results 
the new difference equation 

x k+1 = {A + BF)x k . (1.20) 

Since for any k 

x k+2 = (A + BF) 2 x k , (1.21) 

it follows that, if one finds gains /i and ji with the property that the matrix 
A + BF is nilpotent, that is, 

(A+ BF) 2 = 0, (1.22) 

then we would have a controller with the property that after two sampling steps 
necessarily x k +2 = 0. That is, both ip and tp vanish after these two steps, and 
the system remains at rest after that. This is the objective that we wanted to 
achieve all along. We now show that this choice of gains is always possible. 
Consider the characteristic polynomial 

det{zl- A- BF) = z 2 + (-2cosh£- / 2 sinh S- f x cosh 5+ f±)z 

- /i cosh 5 + 1 + /i + f 2 sinh 6 . (1.23) 

It follows from the Cayley-Hamilton Theorem that condition (1.22) will hold 
provided that this polynomial reduces to just z 2 . So we need to solve for the 
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fi the system of equations resulting from setting the coefficient of z and the 
constant term to zero. This gives 

, 2 cosh (5—1 „ ,2 cosh 5 + 1 

^-^^PT and /2 = - 1/2 ^h^- (L24) 

We conclude that it is always possible to find a matrix F as desired. In 
other words, using sampled control we have been able to achieve stabilization 
of the system. Moreover, this stability is of a very strong type, in that, at least 
theoretically, it is possible to bring the position and velocity exactly to zero in fi- 
nite time, rather than only asymptotically as with a continuous-time controller. 
This strong type of stabilization is often called deadbeat control; its possibility 
(together with ease of implementation and maintenance, and reliability) consti- 
tutes a major advantage of digital techniques. 

1.4 Feedback Versus Precomputed Control 

Note that the first solution that we provided to the pendulum control problem 
was in the form of a feedback law (1.6), where u(t) could be calculated in terms of 
the current position and velocity, which are "fed back" after suitable weighings. 
This is in contrast to "open-loop" design, where the expression of the entire 
control function «(•) is given in terms of the initial conditions ip(0),ip(0), and 
one applies this function u(-) blindly thereafter, with no further observation of 
positions and velocities. In real systems there will be random perturbations that 
are not accounted for in the mathematical model. While a feedback law will 
tend to correct automatically for these, a precomputed control takes no account 
of them. This can be illustrated by the following simple examples. 

Assume that we are only interested in the problem of controlling (1.2) when 
starting from the initial position y(0) = 1 and velocity </J(0) = —2. Some trial 
and error gives us that the control function 

u(t)=3e~ 2t (1.25) 

is adequate for this purpose, since the solution when applying this forcing term 
is 

<p{t) = e~ 2t . 

It is certainly true that ip(t) and its derivative approach zero, actually rather 
quickly. So (1.25) solves the original problem. If we made any mistakes in 
estimating the initial velocity, however, the control (1-25) is no longer very 
useful: 

Exercise 1.4.1 Show that if the differential equation (1.2) is again solved with 
the right-hand side equal to (1.25) but now using instead the initial conditions 

<p(0) = l, ¥>(0) = -2 + £, 
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where £ is any positive number (no matter how small), then the solution satisfies 

lim ip(t) = +00 . 

t — >+oo 

If £ < 0, show that then the limit is —00. □ 

A variation of this is as follows. Suppose that we measured correctly the ini- 
tial conditions but that a momentary power surge affects the motor controlling 
the pendulum. To model this, we assume that the differential equation is now 

<p(t)-ip(t)=u(t)+d(t), (1.26) 

and that the disturbance d(-) is the function 

£ if te [1,2] 



1 otherwise. 
Here e is some positive real number. 

Exercise 1.4.2 Show that the solution of (1.26) with initial conditions <p(0) = 
1,(^(0) = —2, and u chosen according to (1.25) diverges, but that the solution 
of 

(p{t) - <p(t) = -a<p(t) - 0<p(t) + d(t) (1.27) 

still satisfies the condition that <p and ip approach zero. □ 

One can prove in fact that the solution of equation (1.27) approaches zero 
even if d(-) is an arbitrary decaying function; this is an easy consequence of 
results on the input/output stability of linear systems. 

Not only is it more robust to errors, but the feedback solution is also in this 
case simpler than the open-loop one, in that the explicit form of the control as 
a function of time need not be calculated. Of course, the cost of implementing 
the feedback controller is that the position and velocity must be continuously 
monitored. 

There are various manners in which to make the advantages of feedback 
mathematically precise. One may include in the model some characterization 
of the uncertainty, for instance, by means of specifying a probability law for a 
disturbance input such as the above d(-). In any case, one can always pose a 
control problem directly as one of finding feedback solutions, and we shall often 
do so. 

The second solution (1.12)-(1.19) that we gave to the pendulum problem, 
via digital control, is in a sense a combination of feedback and precomputed 
control. But in terms of the sampled model (1.18), which ignores the behavior 
of the system in between sampling times, digital control can be thought of as 
a purely feedback law. For the times of interest, (1.19) expresses the control in 
terms of the "current" state variables. 
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1.5 State-Space and Spectrum Assignment 

We now have seen two fundamentally different ways in which to control the 
linearized inverted pendulum (1.2). They both involve calculating certain gains: 
a and f3 in the case of the continuously acting PD controller (1.6), or the entries 
/i and ji of F for sampled feedback. In both cases we found that appropriate 
choices can be made of these coefficients that result, for instance, in decaying 
exponential behavior if (1.9) holds, or deadbeat control if (1-24) is used. 

That such choices are possible is no coincidence. For very general linear 
systems, it is possible to obtain essentially arbitrary asymptotic behavior. One 
of the basic results in control theory is the Pole-Shifting Theorem, also called 
the "Pole- Assignment" or the "Spectrum Assignment" Theorem, which makes 
this fact precise. 

In order to discuss this Theorem, it is convenient to use the state-space for- 
malism. In essence, this means that, instead of studying a high-order differential 
equation such as (1.1) or (1.2), we replace it by a system of first-order differen- 
tial equations. For instance, instead of (1.2) we introduce the first-order vector 
equation 

x{t) = Ax(t) + Bu(t) , (1.28) 

where x(i) is the column vector (ip(t) , <p(t))' (prime indicates transpose), and 
where A and B are the matrices 

:;).*-(?). a--) 

An equation such as (1-28) is an example of what is called a linear, continuous- 
time, time-invariant, finite dimensional control system. 

The term "continuous-time" refers to the fact that the time variable t is 
continuous, in contrast to a system defined by a difference equation such as 
(1.18). The terminology "time-invariant" is used to indicate that the time t does 
not appear as an independent variable in the right-hand side of the equation. 
If the mass of the pendulum were to change in time, for instance, one might 
model this instead through a time- varying equation. "Finite dimensional" refers 
to the fact that the state x(t) can be characterized completely at each instant 
by a finite number of parameters (in the above case, two). For an example 
of a system that is not finite dimensional, consider the problem of controlling 
the temperature of an object by heating its boundary; the description in that 
case incorporates a partial differential equation, and the characterization of the 
state of the system at any given time requires an infinite amount of data (the 
temperatures at all points). 

If instead of the linearized model we had started with the nonlinear differ- 
ential equation (1.1), then in terms of the same vector x we would obtain a set 
of first-order equations 

x(t) = f(x(t),u(t)), (1.30) 
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where now the right-hand side / is not necessarily a linear function of x and 
u. This is an example of what we will later call a nonlinear continuous-time 
system. 

Of course, we are interested in systems described by more than just two 
state variables; in general x(t) will be an n-dimensional vector. Moreover, often 
there is more than one independent control acting on a system. For instance, 
in a three-link robotic arm there may be one motor acting at each of its joints 
(the "shoulder," "elbow," and "wrist"). In such cases the control u(i) at each 
instant is not a scalar quantity but an m-dimensional vector that indicates the 
magnitude of the external forces being applied at each input channel. Mathe- 
matically, one just lets / in (1.30) be a function of n + m variables. For linear 
systems (1.28) or (1.18), one allows A to be an n x n matrix and B to be a 
rectangular matrix of size n x m. 

The transformation from a high-order single equation to a system of first- 
order equations is exactly the same as that often carried out when establishing 
existence results for, or studying numerical methods for solving, ordinary dif- 
ferential equations. For any a, the vector x(a) contains precisely the amount of 
data needed in order to solve the equation for r > er, assuming that the control 
u(-) is specified for r > a. The state x(a) thus summarizes all the information 
about the past of the system needed in order to understand its future behavior, 
except for the purely external effects due to the input. 

Later we reverse things and define systems through the concept of state. We 
will think of (1-28) not as a differential equation but rather, for each pair of 
times r > a, as a rule for obtaining any value x(t) from the specifications of 
x(a) and of the restriction lu of the control u(-) to the interval between a and 
r. We will use the notation 

x (t) = 4>{t, a, x, u>) (1.31) 

for this rule. We may read the right-hand side as "the state at time r resulting 
from starting at time a in state x and applying the input function w." Because 
solutions of differential equations do not exist in general for all pairs of initial 
and final times a < t, typically 4> may be defined only on a subset of the set of 
possible quadruples (r, <7, X,u>). 

An advantage of doing things at this level of abstraction is that many ob- 
jects other than those described by ordinary differential equations also are rep- 
resentable by a rule such as (1.31). For example, discrete-time equations (1-18) 
also give rise to systems in this sense, since x^ +i depends only on 

Xk,Vk,Vk+l,- ■ ■ ,Vk+i-l 

for all k and i. The idea of using an abstract transition mapping (f> originates 
with the mathematical theory of dynamical systems. The difference with that 
theory is that here we are interested centrally in understanding the effect of 
controls. 
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Returning to the issue of feedback, note that using the matrix formalism 
(1.28) and denoting by F the vector of gains 

F = (-a -/3) 

we may write the PD feedback law (1.6) as 

u{t) = Fx{t). (1.32) 

When we substitute this into (1.28), we obtain precisely the closed-loop equa- 
tions (1.7). So the characteristic equation (1.8) of the closed-loop system is the 
determinant of the matrix 

z -1 

a — 1 z + j3 

that is, the characteristic polynomial of A + BF. The zeros of the characteristic 
equation are the eigenvalues of A + BF. 

We conclude from the above discussion that the problems of finding gains 
a, f3 for the PD controller with the properties that all solutions of (1.7) converge 
to zero, or that all solutions converge with no oscillation, are particular cases 
of the problem of finding a row vector F so that A + BF has eigenvalues with 
certain desired properties. This is precisely the same problem that we solved 
for the discrete-time system (1.18) in order to obtain deadbeat control, where 
we needed A + BF to have a double eigenvalue at zero. Note that in the second 
case the matrices A and B are different from the corresponding ones for the 
first problem. Note also that when dealing with vector-input systems, for which 
u(t) is not a scalar, the F in equation (1.32) must be a matrix instead of a row 
vector. 

Thus one arrives at the following purely algebraic question suggested by both 
the PD and the deadbeat digital control problems: 

Given a pair of real matrices (A,B), where A is square of size nxn 
and B is of size nxm, characterize the set of all possible eigenvalues 
of A + BF as F ranges over the set of all possible real matrices of 
size m x n. 

The Spectrum Assignment Theorem says in essence that for almost any pair 
of matrices A and B it is possible to obtain arbitrary eigenvalues for A + BF 
using suitable feedback laws F, subject only to the obvious constraint (since 
A + BF is a real matrix) that complex eigenvalues must appear in conjugate 
pairs. "Almost any" means that this will be true for all pairs that describe what 
we will call controllable systems, those systems for which it is possible to steer 
any state to any other state. We will see later that controllability corresponds 
to a simple nondegeneracy condition relating A and B; when there is just one 
control (m= 1) the condition is simply that B, seen as a column vector, must 
be cyclic for A. 
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This Theorem is most often referred to as the Pole-Shifting Theorem, a 
terminology that is due to the fact that the eigenvalues of A + BF are also the 
poles of the resolvent operator 

(zl- A-BF)- 1 : 

or equivalently, of the rational function l/det(z/ — A— BF); this rational func- 
tion appears often in classical control design. 

As a consequence of the Pole-Shifting Theorem we know that, save for very 
exceptional cases, given any system 

x(t) = Ax(t) + Bu{t) 

it will be possible to find a feedback law u(t) = Fx(t) such that all solutions of 
the closed-loop equation 

x(t) = (A + BF)x{t) 

decay exponentially. To conclude this, one uses the Pole-Shifting Theorem to 
obtain a matrix F so that all eigenvalues of A + BF are negative real numbers. 
Another illustration is the general deadbeat control problem: Given any dis- 
crete-time system (1.18), one can in general find an F so that (A + BF) n = 0, 
simply by assigning zero eigenvalues. Again, a nondegeneracy restriction will 
be needed, as illustrated by the counterexample A = I, B = for which no such 
F exists. 

The Pole-Shifting Theorem is central to linear systems theory and is itself the 
starting point for more interesting analysis. Once we know that arbitrary sets of 
eigenvalues can be assigned, it becomes of interest to compare the performance 
of different such sets. Among those that provide stability, some may be more 
useful than others because they have desirable transient characteristics such as 
lack of "overshoot." Also, one may ask what happens when certain entries of 
F are restricted to vanish, which corresponds to constraints on what can be 
implemented. 

Yet another possibility is to consider a cost criterion to be optimized, as 
we do when we discuss optimal control theory. For example, it is possible to 
make trajectories approach the origin arbitrarily fast under the PD controller 
(1.6), but this requires large gains a,/3, which means that the controls u(t) to 
be applied will have large magnitudes. Thus, there is a trade-off between the 
cost of controlling and the speed at which this may be achieved. 

In a different direction, one may investigate to what extent the dynamics of 
nonlinear and/or infinite dimensional systems are modifiable by feedback, and 
this constitutes one of the main areas of current research in control theory. 

Classical Design 

The next exercise on elementary ordinary differential equations is intended to 
convey the flavor of some of the simpler performance questions that might be 
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asked when assigning eigenvalues and to mention some standard terminology. 
Techniques from the theory of functions of a complex variable are often used in 
this type of study, which is typically the subject of undergraduate engineering 
courses in the "classical design" or "frequency design" of control systems. We 
include this discussion as an illustration but will not treat frequency design in 
this text. 

Exercise 1.5.1 Consider again the pendulum linearized about its unstable up- 
per position, given by the equation (pit) — ip(t) = u(t), and assume that we use 
the PD control law u(t) = — cnp{i) — (3<p(t) to obtain an asymptotically stable 
closed-loop system ip(t) + bcp(t) + acp(t) = (with a = a — 1 > and b = /3 > 0). 
Introduce the natural frequency uj := yfa and the damping factor Q := 6/(2y / a), 
so that the equation now reads 

<p(t) + 2(wip(t) + w 2 ip(t) = . (1.33) 

(A) Prove the following facts: 

1. If £ < 1 (the "underdamped" case), all solutions are decaying oscillations. 

2. If £ = 1 (the "critically damped" case) or if £ > 1 ( "overdamped" ) , then 
all solutions for which </?(0) 7^ are such that ip(t) = for at most one 
t > 0. 

3. If £ > 1, then every solution that starts from rest at a displaced position, 
that is, <p(0) 7^ and <p(0) = 0, approaches zero monotonically. (In this 
case, we say that there is no "overshoot.") 

4. Show rough plots of typical solutions under the three cases £ < 1, £ = 1, 
and C > 1. 

(B) Consider again the underdamped case. Using rough plots of the solutions 
of the equation with </?(0) = — 1, <f(0) = both in the case when £ is near zero 
and in the case when £ is near 1, illustrate the fact that there is a trade-off 
between the speed of response (how fast one gets to, and stays near, the desired 
value for ip) and the proportion of overshoot (the maximum of ip{t) given the 
initial magnitude 1). Show that for £ = V2/2 this overshoot is e~ ,r < 0.05. 

(C) Assume now that the objective is not to stabilize the system (1.2) about 
<p = ip = 0, but to make it assume some other desired value <p = ifd, still with 
velocity (p approaching zero. If we were already at rest at this position tp = ip<j, 
then the constant control u = — ipd would keep us there. Otherwise, we must 
add to this a correcting term. We then modify the PD controller to measure 
instead the deviation from the desired value ipd'- 

u(t) = -a((p(i) - (fid) - P<p(i) - <Pd ■ 

Show that with this control, and still assuming a > 1, (3 > 0, the solutions of the 
closed- loop system indeed approach ip = ip^, ip = 0. One says that the solution 
has been made to track the desired reference value ipd- □ 
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The value C = V^/S appears often in engineering practice, because this 
results in an overshoot of less than 5%, a figure considered acceptable. If we 
write 9 for the angle between the negative real axis and the vector z, where z 
is either root of the characteristic equation 

z 2 + 2C,u)z + uj 2 = , 

then £ = cos 9. Thus, a value 9 = 45° or less is often sought. If in addition one 
wants the magnitude of solutions to decay at a given exponential rate, say, as 
e _At , then one looks at placing eigenvalues in a region that forces the desired 
stability degree, such as that illustrated in Figure 1.4, which exhibits a "damping 
margin" of 45° and a "stability margin" of A. 



Figure 1.4: Typical desired eigenvalue locations. 

Because solutions of more general linear equations can be written as sums 
of those corresponding to complex pairs and real roots, designers often apply 
these same rules to systems of order higher than two. 



1.6 Outputs and Dynamic Feedback 

We introduce now another major component of control theory, the idea of mea- 
surements and outputs. 

In some situations all of the variables needed in order to implement a feed- 
back control are readily available, but other times this is not the case. For 
instance, the speed of a car can be obtained easily via a speedometer, but the 
ground speed of an aircraft may be difficult to estimate. To model mathemati- 
cally such a constraint, one adds to the description of a system the specification 
of a measurement function. This is a function of the states, with scalar or vector 
values, which summarizes all of the information that is available to the control 
algorithm. 

Returning once more to the inverted pendulum example, let us assume that 
in designing the PD controller (1.6) we are restricted to measuring directly only 
the angular position f(t). We model this by adding to the state space equation 
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(1.28), where A and B are given by (1.29), the mapping 

C :M. 2 ^M. : x i-> aji , (1.34) 

or equivalently its matrix in the canonical basis, C = (1,0), that picks the 
variable (p. We write y(t) for the allowed measurement at time t, that is, 

y(t) = Cx(t). (1.35) 

More generally, we define a linear system with output by giving triples of matrices 
(A, B, C), where for some integers n, m,p, A is n x n, B is n x m, and C is pxn. 
The integer p can be thought of as the number of independent measurements 
that can be taken at each instant. One also defines nonlinear and discrete-time 
systems with output in an analogous way, or more abstractly by adding an 
output map to the axiomatic definition of system via transitions (1.31). 

A linear feedback law that depends only on the allowed measurements is 
then, in terms of this notation, of the form 

u{t) = KCx{t). (1.36) 

In other words, F is restricted to factor as F = KC ', where C is given. As dis- 
cussed when dealing with proportional-only control, such static output feedback 
is in general inadequate for control purposes. The set of eigenvalues achievable 
for matrices A + BKC by varying K is severely restricted. 

Of course, there is no theoretical reason for restricting attention to static 
feedback laws. A controller that would incorporate a device for differentiating 
the observed position x\{i) = tp(t) could then use this derivative. 

However, differentiation tends to be an undesirable operation, because it is 
very sensitive to noise. To illustrate this sensitivity, consider the problem of 
finding the derivative c of the function 

S(t)=ct, 

where c is an unknown constant. Assume that the data available consists of £(£) 
measured under additive noise, 

y(t) = £(*) + n(t) , 

where n{t) is of the form 

n(t) = rfsin ut , 

and d and w are unknown constants, to being large. (This is realistic in that 
noise effects tend to be of "high frequency" compared to signals, but of course in 
real applications the noise cannot be modeled by a deterministic signal of con- 
stant frequency, and a probabilistic description of n(t) is needed for an accurate 
analysis.) If we simply differentiate, the result 

y(t) = c + duj cos tot 
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can be very far from the desired value c since to is large. An alternative is to 
use the fact that the noise n(t) averages to zero over long intervals; thus, we 
can cancel its effect by integration. That is, if we compute 

ft 
y(r) dr 



2 (l-coswt), (1.37) 



and we then multiply this by 2/t 2 , we obtain 

2d 

c 

uit 

which converges to c as t — > oo and hence provides an accurate estimate, asymp- 
totically, of the desired derivative. 

The topic of observers deals with obtaining estimates of all the state vari- 
ables from the measured ones, using only integration — or, in discrete-time, 
summations. It is an easy consequence of general theorems on the existence of 
observers that one can achieve also stabilization of the inverted pendulum, as 
well as of most other continuous-time linear systems, using a controller that is 
itself a continuous-time linear system. Such a controller is said to incorporate 
integral or dynamic feedback, and it includes a differential equation, driven by 
the observations, that calculates the necessary estimate. Analogously, for dis- 
crete-time systems with partial observations, the theorems result in the existence 
of observers that are themselves discrete-time linear systems. 

In particular, there exist for the pendulum coefficients a,/3,fi,i>, so that 

z(t) = vz(i) + nxi(i) (1.38) 

together with a feedback law 

u(t) = -axi(t) - (3z(t) (1.39) 

stabilizes the system. The controller solves the differential equation (1.38) with 
an arbitrary initial condition z(0) and feeds the linear combination (1.39) of 
the measured variable X\ and the estimated variable z back into the system. 
Later we develop in detail a systematic approach to the construction of such 
dynamic controllers, but in this simple example it is easy to find the necessary 
parameters by analyzing the equations. 

Exercise 1.6.1 Consider the system of three differential equations obtained 
from (1.28) together with (1.38), where u(t) is given by (1.39). Find numbers 
a, P, [i, v, such that all solutions of this system approach zero. □ 

When observations are subject to noise, the speed at which one can estimate 
the unmeasured variables — and hence how fast one can control — is limited by 
the magnitude of this noise. For instance, the convergence of the expression in 
equation (1.37) is slower when d is large (noise has high intensity) or lu is small 
(noise has large bandwidth). The resulting trade-offs give rise to problems of 
stochastic filtering or stochastic state estimation, and the Kalman Filter is a 
technique for the analysis of such problems. 
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PID Control 

Dynamic control is also useful in order to cancel unwanted disturbances even if 
all state variables are available for feedback. We illustrate this using again the 
inverted pendulum model. Assume that an unknown but constant disturbance 
d(t) = e can act on the system, which is therefore described by 

ip(t) -ip(t) = u(t) + e. (1.40) 

One might still try to use the PD controller (1.6) to stabilize this system inde- 
pendently of e. Since stability should hold in particular when e = 0, one should 
have a > 1 and f3 > as before. But the PD controller is not sufficient: 

Exercise 1.6.2 Show that if e ^ and a > 1, /? > 0, then no solution of 

<p(t) + 0<p{t) + (a - l)(p{t) = e 

is such that (p(t) — > as t — > oo. □ 

Since e is a constant, it can be thought of as the solution of the differential 
equation 

£o(t)=0. (1.41) 

If we think of xq as a quantity that cannot be measured, then the previous 
discussion about observers suggests that there may exist a dynamic controller: 

Exercise 1.6.3 Find numbers a,0,fj, with the following property: For each 
e G R, all of the solutions of the system of equations 



x (t) = 


= Xl(t) 


±l(t) = 


= x 2 (t) 


x 2 (t) = 


= -/xx (t) 



+ (1 - a)xi(t) - Px 2 (i) + e 

converge to (e//i,0,0). (Hint: First solve this problem for the homogeneous 
system that results if e = by finding parameters that make the matrix associ- 
ated to the equations have all eigenvalues with negative real parts. Then show 
that the same parameters work in the general case.) □ 

Thus, using a controller that integrates x\(t) = tp(t) and feeds back the 
combination 

u (t) = -a(p(t) - P<p(t) - n / (p(T)dT (1.42) 

Jo 

one can ensure that both <p and ip converge to zero. Another way of think- 
ing of this controller is as follows: If the integral term is not used, the value 
ip(t) approaches a constant steady-state error, the effect of the integral is to 
offset a nonzero error. The controller (1.42) is a PID, or proportional-integral- 
derivative feedback, the control mechanism most widely used in linear systems 
applications. 
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More generally, the problem of canceling the effect of disturbances that are 
not necessarily constant gives rise to the study of disturbance rejection or dis- 
turbance decoupling problems. The resulting dynamic controllers typically in- 
corporate an "internal model" of the possible mechanism generating such dis- 
turbances, as with the simple case (1.41). 

Observability, Duality, Realization, and Identification 

A basic system theoretic property that arises naturally when studying observers 
and dynamic feedback is that of observability. Roughly speaking, this means 
that all information about the state x should in principle be recoverable from 
knowledge of the measurements y that result; in a sense it is a counterpart to the 
concept of controllability — mentioned earlier — which has to do with finding 
controls u that allow one to attain a desired state x. In the case of linear 
systems, this analogy can be made precise through the idea of duality, and it 
permits obtaining most results for observers as immediate corollaries of those 
for controllability. This duality extends to a precise correspondence between 
optimal control problems — in which one studies the trade-off between cost of 
control and speed of control — and filtering problems — which are based on the 
trade-off between magnitude of noise and speed of estimation. 

Once outputs or measurements are incorporated into the basic definition of a 
system, one can pose questions that involve the relationship between controls u 
and observations y. It is then of interest to characterize the class of input/output 
(i/o) behaviors that can be obtained and conversely to study the realization 
problem: "Given an observed i/o behavior, what possible systems could give 
rise to it?" In other words, if we start with a "black box" model that provides 
only information about how u affects y, how does one deduce the differential — 
or, in discrete-time, difference — equation that is responsible for this behavior? 

Besides obvious philosophical interest, such questions are closely related to 
identification problems, where one wishes to estimate the i/o behavior itself 
from partial and possibly noisy data, because state-space descriptions serve to 
parametrize such possible i/o behaviors. Conversely, from a synthesis viewpoint, 
realization techniques allow us to compute a state representation, and hence also 
construct a physical system, that satisfies given i/o specifications. 

The main results on the realization problem show that realizations essen- 
tially are unique provided that they satisfy certain minimality or irredundancy 
requirements. We will provide the main theorems of realization theory for lin- 
ear systems. The underlying properties turn out to be closely related to other 
system theoretic notions such as controllability and observability. 



1.7 Dealing with Nonlinearity 

Ultimately, linear techniques are limited by the fact that real systems are more 
often than not nonlinear. As discussed above, local analysis and control design 
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in general can be carried out satisfactorily using just first-order information, 
but global analysis requires more powerful methods. 

One case in which global control can be achieved with ideas from linear 
system theory is that in which the system can be reduced, after some trans- 
formation, to a linear one. For instance, a change of coordinates in the space 
of the x variable might result in such a simplification. A situation that has 
appeared many times in applications is that in which a particular feedback con- 
trol linearizes the system. Consider again the pendulum, but now the original 
nonlinear model. 

If we first subtract the effect of the term sin 9 by using u, we are left with a 
simple linear system 

9(t)=u(t), (1.43) 

which can be controlled easily. For instance, the PD feedback law 

u(t) = -0{t) - 6{t) (1.44) 

stabilizes (1.43). In order to stabilize the original system, we now add the 
subtracted term. That is, we use 

u(t) = sin 9{t) - 9{t) - 9{t) . (1.45) 

Passing from the original model to (1-43) can be thought of as the effect of 
applying the feedback law 

u(t) = f(x(t)) + u'(t), 

where f(x) = sinxi and u' is a new control; the study of such feedback trans- 
formations and the characterization of conditions under which they simplify the 
system is an active area of research. 

The above example illustrates another issue that is important when dealing 
with the global analysis of nonlinear control problems. This is the fact that the 
mathematical structure of the state space might impose severe restrictions on 
what is achievable. The proposed feedback law (1.45) will stabilize 

9(t) +sm6(t) = u(t) 

provided we can think of 9 as taking arbitrary real values. However, in the 
physical system 8+2tt describes the same state as 9. Thus, the natural space to 
describe 9 is a unit circle. When applying the control u as defined by the formula 
(1.45), which of the infinitely many possible values of 9 should one use? It turns 
out that everything will behave as desired as long as one chooses u continuously 
in time — as the pendulum finishes one complete counterclockwise revolution, 
start using an angle measured in the next 27r-interval. But this choice is not 
unambiguous in terms of the physical coordinate 9. In other words, (1-45) is not 
a feedback law when the "correct" state space is used, since it is not a function 
of states. 
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The correct state space for this example is in fact the Cartesian product 
of a circle and the real line, since pairs (8(t),6(t)) belong naturally to such a 
space. It can be proved that in such a space — mathematically the tangent 
bundle to the unit circle — there cannot exist any continuous feedback law that 
stabilizes the system, because the latter would imply that the space is diffeomor- 
phic (smoothly equivalent) to a Euclidean space. In general, techniques from 
differential geometry — or, for systems described by polynomial nonlinearities, 
algebraic geometry — appear when studying global behavior. 

When dealing with the problems posed by infinite dimensionality as opposed 
to nonlinearity, similar considerations force the use of techniques from functional 
analysis. 

1.8 A Brief Historical Background 

Control mechanisms are widespread in nature and are used by living organisms 
in order to maintain essential variables such as body temperature and blood 
sugar levels at desired setpoints. In engineering too, feedback control has a long 
history: As far back as the early Romans, one finds water levels in aqueducts 
being kept constant through the use of various combinations of valves. 

Modern developments started during the seventeenth century. The Dutch 
mathematician and astronomer Christiaan Huygens designed pendulum clocks 
and in doing so analyzed the problem of speed control; in this work he competed 
with his contemporary Robert Hooke. The needs of navigation had a strong 
influence on scientific and technological research at that time, and accurate 
clocks — to allow determinations of solar time — were in great demand. The 
attention turned to windmills during the eighteenth century, and speed controls 
based on Hooke's and Huygens' ideas were built. A central idea here is the use 
of flyballs: Two balls attached to an axis turn together with the windmill, in 
such a manner that centrifugal force due to angular velocity causes them to rise; 
in turn this upward movement is made to affect the positions of the mill's sails. 
Thus, feedback was implemented by the linkages from the flyballs to the sails. 

But it was the Industrial Revolution, and the adaptation by James Watt in 
1769 of flyball governors to steam engines, that made control mechanisms very 
popular; the problem there was to regulate engine speed despite a variable load. 
Steady-state error could appear, and various inventors introduced variations of 
the integral feedback idea in order to deal with this problem. 

The mathematician and astronomer George Airy was the first to attempt, 
around 1840, an analysis of the governor and similar devices. By 1868, there 
were about 75,000 Watt governors in use; that year, the Scottish physicist James 
Clerk Maxwell published the first complete mathematical treatment of their 
properties and was able to explain the sometimes erratic behavior that had been 
observed as well as the effect of integral control. His work gave rise to the first 
wave of theoretical research in control, and characterizations of stability were 
independently obtained for linear systems by the mathematicians A. Hurwitz 
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and E.J. Routh. This theory was applied in a variety of different areas, such as 
the study of ship steering systems. 

During the 1930s, researchers at Bell Telephone Laboratories developed the 
theory of feedback amplifiers, based on assuring stability and appropriate re- 
sponse for electrical circuits. This work, by H. Nyquist, H.W. Bode, and others, 
constitutes even today the foundation of much of frequency design. Analog 
computers appeared also around that time and formed the basis for implement- 
ing controllers in the chemical and petroleum industries. During the Second 
World War, these techniques were used in the design of anti-aircraft batteries 
and fire-control systems, applications that continue to be developed today. The 
mathematician Norbert Wiener, who developed a theory of estimation for ran- 
dom processes used in these applications, coined at the end of the war the term 
"cybernetics" to refer to control theory and related areas. 

These so-called classical approaches were for the most part limited by their 
restriction to linear time-invariant systems with scalar inputs and outputs. Only 
during the 1950s did control theory begin to develop powerful general techniques 
that allowed treating multi variable, time- varying systems, as well as many non- 
linear problems. Contributions by Richard Bellman (dynamic programming) 
and Rudolf Kalman (filtering, linear/quadratic optimal control, and algebraic 
analysis) in the United States, and by L. Pontryagin (nonlinear optimal control) 
in the Soviet Union, formed the basis for a very large research effort during the 
1960s, which continues to this day. Present day theoretical research in control 
theory involves a variety of areas of pure mathematics, as illustrated for instance 
by the remarks in Section 1.7. Concepts and results from these areas find appli- 
cations in control theory; conversely, questions about control systems give rise 
to new open problems in mathematics. 

Excellent references for the early historical background are the papers [149], 
which contain a large number of literature citations, and the book [299]. See 
also the introductory article [226]. Other references, which in addition contain 
overviews of current research and open problems, are the reports [275] and [135]. 



1.9 Some Topics Not Covered 

In an area as wide as control theory, it is impossible to cover all topics in a single 
text, even briefly. For instance, we concentrate exclusively on deterministic 
systems. Incorporating models for uncertainty leads to stochastic models, which 
are the subject of much research activity, when this uncertainty is expressed 
in probabilistic or statistical terms; references to the literature on stochastic 
aspects of many of the problems considered are given at various points in the 
text. Mathematically different but closely related is the area of robust control, 
which deals with the design of control laws that are guaranteed to perform even 
if the assumed model of the system to be controlled is incorrect — with the 
allowed deviations quantified in appropriate norms — or under the possibility 
of imperfect controller design. See the collection of papers [123] for pointers to 
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a large literature. The area of adaptive control deals also with the control of 
partially unknown systems, but differs from robust control in the mathematics 
employed. Adaptive controllers typically make use of identification techniques, 
which produce estimates of the system parameters for use by controllers; see 
e.g. [282]. 

When using computers one should consider the effect of quantization errors 
on the implementation of control laws, which arise due to limited precision 
when real- valued signals are translated into fixed-point representations (A/D 
or analog to digital conversion); see e.g. [304]. Other questions relate to the 
interface between higher-level controllers implemented in software and lower- 
level servos of an analog and physical character; this gives rise to the area of 
hybrid systems; see the volume [15]. 

The notes at the end of the various chapters give further bibliographical 
references to many of the other areas that have been omitted. 



Chapter 2 

Systems 



This Chapter introduces concepts and terminology associated with discrete- 
and continuous-time systems, linearization, and input/output expansions, and 
establishes some of their elementary properties. 



2.1 Basic Definitions 

As seen in the previous Chapter, it is of interest to study both discrete-time 
and continuous-time systems. In the first case, the underlying time set is dis- 
crete, and it can be thought of as the set of integers Z; in the second case the 
time variable is a real number. To treat both simultaneously, we introduce the 
following notion. In most of our results it will be the case that either T = Z or 



Definition 2.1.1 A time set T is a subgroup o/(K, +). □ 

For any such set, T + is the set of nonnegative elements {t € T\t > 0}. By 
notational convention, when the time set T is understood from the context, all 
intervals are assumed to be restricted to T. Thus, for instance, 

[a,b) = {t e T,a < t < b} 

and similarly for open, closed, or infinite intervals. 

For each set IX and interval X, the set of all maps from X into IX is denoted 
by 

U x = {w| w:X->U}. (2.1) 

If T = Z and k is a nonnegative integer, the set UX°' fe ) can be identified naturally 
with the set of all sequences 

w(0),...,w(fc-l) 

25 
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of length k consisting of elements of U, i.e. the Cartesian product li k . In 
the particular case in which X is an empty interval, the set in (2.1) consists of 
just one element, which we denote as o; this can be thought of as the "empty 
sequence" of zero length. 

Assume that a, r, /z, are any three elements of T that satisfy a < r < (/,. If 
lo\ G w- a ' r ' and lo 2 G w- t '^\ their concatenation, denoted simply as lo\lo 2 , is the 
function lo G XV-' 7 -^ defined by 



y ' \ uj 2 {t) if te \t,^). 



Note that ojo = ou-> = w. 

The next definition provides the abstraction of the concept of system that 
was discussed on page 12. 

Definition 2.1.2 A system or machine E = (T, X, It, (/)) consists of: 

• A time set T ; 

• A nonempty set X called the state space o/S; 

• A nonempty set U called the control- value or input-value space o/S; 
and 

• ^4 map 4> '■ T^rj> — * X called the transition map o/S, which is defined on 
a subset T>^ of 

{(t, a, x, to) | a, t G T, a < r, x G X, lo G U [o "' r) } , (2.2) 

such that the following properties hold: 



nontriviality For each state x G X, there is at least one pair a < t in T 

and some lo G r \v- a ' T > such that to is admissible for x, that is, so that 

(t,o-,x,lo) g £>0; 



restriction 



If to a 11^^' is admissible for x, then for each r G [a, /J,) the 
restriction t0\ := w|u T ) of lo to the subinterval [a, r) is also admissible for 
x and the restriction lo 2 := w|r T „) is admissible for 4>(t,o~,x,loi); 



semigroup 



// (7, r, fi are any three elements of T so that a < T < (X, if 



lo\ G 1v- a ' T > and lo 2 G W- T '^' , and if x is a state so that 

(/)(t, a, x, wi) = X\ and <p(fi,r, xi,to 2 ) = x 2 , 
then to = lo\lo 2 is also admissible for x and 4>(fi, a, X, lo) = x 2 ; 



identity For each a G T and each x G X, the empty sequence o G Vo-"^' is 



admissible for x and 0(er, a, x, o) = x. D 
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As discussed in Chapter 1, Section 1.6, one often needs to model explicitly 
the fact that limited measurements are available. The following concept is then 
natural. 

Definition 2.1.3 A system or machine with outputs is given by a system E 
together with 

• A set V called the measurement- value or output-value space; and 

• A map h : T x X — > y called the readout or measurement map. □ 

We use the same symbol for the system with outputs (T, X, LI, <j), V, h) and 
its underlying system (T, X, It, </>). Thus E could denote either the first or 
the second of these, depending on the context. Unless otherwise stated, every 
definition given for systems will apply to systems with outputs by reference 
to its underlying system. For instance, we define in a later section the notion 
of a controllable system; a controllable system with outputs will then be, by 
definition, one whose underlying system is controllable. 

Elements of X are called states, elements of IX are control values or input 
values, and those of y are output values or measurement values. The 
functions u G VS a ' T > are called controls or inputs. If lo is like this, with a and 
r in T, one denotes by 

\u>\ = t — a 

the length of ui. If lu is admissible for x, we also say that (x,lo) forms an 
admissible pair. 

The definition of system is intended to capture the intuitive notion of a 
machine that evolves in time according to the transition rules specified by (/>. 
At each instant, the state x summarizes all of the information needed in order 
to know the future evolution of the system. For instance, in the case of the 
pendulum we took as the state the pair (angular position, angular velocity). 
We read </>(t, a, x, u>) as "the state at time r resulting from starting at time a in 
state x and applying the input function w." The definition allows the possibility 
of undefined transitions, when the input ui is not admissible for the given state, 
which correspond for differential equations to the phenomenon of "explosion 
times" at which the solution ceases to be defined. 

A few simplifying conventions are useful. When a and r are clear from the 
context, we write 4>(t,o~,x,lu) simply as <p(x,u>). Often, a formula such as 

z = 4>{x,uj) (2.3) 

will implicitly mean "u> is admissible for x and z = 4>(x,lu). v When more 
than one system is being studied, we sometimes add a subscript and write 
Xs, Us, 0s to emphasize that we are considering the set X, It, or the map 
4>, corresponding to the particular system E. Sometimes we refer simply to 
"the system (X, It, <$>)," or even just (X, 4>), when the time set T, or It, is clear 
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from the context, and similarly for systems with outputs. A particularly useful 
notation is the following: We write 

x-^z (2.4) 

instead of equation (2.3). Then the semigroup axiom is simply 

X^X-t and ii^io => x ~> x 2 (2.5) 

and the identity axiom is 

x~^x (2.6) 

for all x. 

It is also convenient to introduce a notation for the mapping tp that assigns 
to each admissible pair (x,u>) the complete state trajectory on [a, t] rather than 
just the final state at time r. That is, 

where 

S(t):=<t>(t,a,x,w\l*,t)) (2-7) 

for each t G [a, r] . Note that the right-hand side of (2.7) is well defined because of 
the restriction axiom. As with <j>, we write simply tp(x, u>) instead of ip(r, a, x, u>) 
when a and r are clear from the context. 

It is useful to define also admissibility for infinite- length controls. 

Definition 2.1.4 Given a system E and a state i£l, the function to € W-' 7 ' 00 ' 
is admissible for x provided that every restriction lo\[ct,t) *- s admissible for x, 
for each t > a . □ 

We also use the term "control" to refer to such an infinite length u), and write 
ip(oo, o~, x, u>) for the function £ € X^' 00 ' which satisfies (2.7) for all t € [c, 00). 
Note that the expression </>(oo, u, x, uj) has no£ been defined; a natural definition, 
however, would be as 

lim ip(oo,<j, x,oj)(t) , 

t — >oo 

and this is basically what will be done later when dealing with asymptotic 
notions, for those systems for which X has a topological structure. 

Definition 2.1.5 A trajectory V for the system, £ on the interval! C T is a 
pair of functions (£, w), £ € X 2 \ui € U 1 such that 

£(r) = ^(r,c7,^(o-),w| [(7ir )) 

ZioZds /or eac/i pair cr, r € X, cr < r. □ 

Definition 2.1.6 ^4 path of the system E on i/ie interval X C T is a function 
£ : X — » X /or which there exists some u> € IX 1 smc/i i/iai (£, w) is a trajectory on 
X for E. If t; is a path and X has the form [cr, r) or [cr, r] (cr finite), then !;(o~) is 
the initial state 0/ i/ie pai/i. 7/X /ias i/ie form (c, r] or [a, t] (t finite), then 
£(t) is f/ie terminal state 0/ £/ie pai/i. □ 
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Some Taxonomy 

Various general properties serve to classify systems into broad classes, as dis- 
cussed in the introductory chapter. We introduce next some of these classes. 
Discrete- and continuous-time systems will be defined in the next section. 

Definition 2.1.7 The system E is complete if every input is admissible for 
every state: 

V 4> = {{t,<t,x,uj) | a<T,x£X,io£U la ' T) }. 

More generally, for any family of controls V C [j IJ<T lv- a ' T \ V-completeness 
means that (r, a, x, uS) is in V<p whenever (7<t,ig1,wgV. □ 

For continuous-time systems (see below) completeness will always mean In- 
completeness, where V is the class of all essentially bounded measurable controls. 

The nontriviality and restriction axioms in the definition of system are auto- 
matically true if the system is complete. In that case, a system is precisely the 
same as an action of U* on the set X, where IX* is the union of the sets W- <7 ' T ', 
thought of as a semigroup with a partially defined binary operation (concate- 
nation). Many of the concepts that we study are direct analogues of those 
studied for semigroup or group actions; for instance, "controllability" will be 
the analogue of "transitivity." 

Systems defined by classical differential or difference equations become "sys- 
tems" in our sense when they are reinterpreted as having a one-element control- 
value set. The closed-loop system that we obtained after designing a feedback 
law for the pendulum was such a system, as would be the unforced pendulum 
that results when setting the control to zero. 

Definition 2.1.8 The system E has no controls if 11 is a one-element set. □ 

We also say then that S is a classical dynamical system. The often-used 
terminology autonomous has the disadvantage of appearing in the theory of 
differential equations to refer to what we call below time-invariance. For systems 
with no controls, there is for each o~,t only one possible element in W (T ' T ' , so 
</>(t, o~, x, to) is independent of the last coordinate; we then write simply cj>(r, a, x). 

The most important general subclass of systems is that of time-invariant 
systems, for which the structure is independent of time, that is, <J)(t,<t,x,u)) 
depends only on r — a, x, to: 

Definition 2.1.9 The system E is time-invariant if for each u) £ w a ' T > , each 
x £ X, and each n £ T, if u is admissible for x then the translation 

^ e U^"^"', u,"(t) := u(t - n) (2.8) 

is admissible for x, and 

<j)(T,a,x,uj) = 4>(t + [i,a + ix, x,!*) 1 *). 
For systems with outputs, it is also required that h(r,x) be independent of t. D 
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When dealing with time-invariant systems, it is customary to identify con- 
trols with their translations. Thus, for instance, given u)\ € V}°> (7 > and LO2 € 
IX [°' T ), one thinks of their concatenation <jJ\Ui as an element of lii°' <T ~ + \ mean- 
ing the concatenation uj\v, where v is the translation of 102 to [a, a + r). A 
system that is not necessarily time-invariant is sometimes called, to emphasize 
that fact, a time- varying system. 

2.2 I/O Behaviors 

Frequently physical systems are specified not by means of their state-space de- 
scriptions but rather through their input/output behavior, that is to say, the 
effect that inputs have on observed outputs. In this section we introduce i/o 
behaviors and show how these are induced by systems with outputs in the sense 
of Definition 2.1.3. 

Technically, there are at least two possible ways of specifying i/o behavior. 
The first, the one to be followed here, is to give a rule that tells us what is 
the output value that results right after an input has been applied. The second 
possibility is to specify the entire output, as a function of time, that is generated 
by this input. 

Definition 2.2.1 An i/o behavior A = (T, II, V, A) consists of: 

• A time set T ; 

• A nonempty set U called the control- value or input- value space of A; 

• A nonempty set y called the output-value or measurement-value 
space of A; and 

• A map 

A : V x -» y 

called the response map of A, which is defined on a nonempty subset T>\ 
of 

{(r,a,u)) \a,T€T,a<T,oj€U [a ' T) }, 

such that the following property holds: 



restriction If (r,a,u>) is in T>\, then also (fi,a,Lu\< afl \) € T>\ for each fi € 
[a,r]. ' □ 

For each fixed a, r, we often denote 

A CT < T M:=A(T,a,co) 

or simply A(w) if a and r are clear from the context. 
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Remark 2.2.2 Given an i/o behavior A, one may define also the i/o map 
associated to A, as follows: This is the map that gives the entire output function 
when a control has been applied, that is, 

A : V x -> |J y^l , 

ct<t 

where 

A(t,ct,w)0) := A(£,cT,u;|[ CTit )) 

for each £ € [<7, r]. Note that the following property holds: 



strict causality If (t,(t,lu) is in T>\, then for each /i € [er, r], 

A(m,ct,w|[ CTiju) ) = A(t,ct,o;)|[ CT)JU ] . 

Note also that the response can be recovered from the i/o map, since 

A(t,ct,u/) = A(t,ct,w)(t). □ 

Our interest will be mainly in those behaviors that arise from systems after 
choosing an initial state. 

Definition 2.2.3 An initialized system is a pair (£, x°), where x° is a state, 
the initial state. Similarly for systems with outputs. □ 

Definition 2.2.4 Let (S,x°) be an initialized system. The input to state 
(i/s) behavior of the system (Y*,x a ) is the behavior with the same sets T 
and It as S, and with output value space X, whose response map is defined on 
the projection 

{(T,a,aj)\(T,a,x ,uj)€V^ (2.9) 

by the formula 

A(t, a, lu) := 4>(t, a, x ,uj) . 

If (T,,x°) is an initialized system with output, then the i/o behavior of (E,x ) 
has the same T, XL, and y as Y,, domain again equal to (2.9), and 

A(r, a, u) := h(r, 4>(t, o-,x a ,u)) . 

This map is the response map of (S,x°). □ 

Subscripts such as in Ag or A s x o may be used to indicate that one is referring 
to the behavior or the response map associated to a given system. 

It is clear from the definitions that both the i/s and the i/o behaviors of a 
system are indeed i/o behaviors. Observe that the i/o map associated to the i/s 
response in the sense of Remark 2.2.2 is precisely the same as ?/>(•, -,x°, •) (cf. 
formula (2.7)). Note also that from the definitions and the identity axiom, 

Ae,x° (o-, o-, o) = h(o-,x°) 
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for all er £ T. 

In general there will be infinitely many systems that induce a given i/o 
behavior, the situation being entirely analogous to that which arises if we take 
the composition of two maps 7 = aj3: Even though 7 is uniquely defined from 
a and /?, the latter are far from being determined just from the knowledge of 7, 
unless stringent hypotheses are made as to their form. This converse issue, that 
of passing from a given i/o behavior to a system with outputs that gives rise to 
the behavior, and the classification of the possible solutions, is the realization 
problem, to be studied in detail later (Sections 6.5 and 6.8). 

The next definition introduces some standard families of behaviors. 

Definition 2.2.5 The i/o behavior A is 

complete ifV x = {(t,<j,u>) | <r, r £ T,a < t,u> £ 1X [o '' t) }; 

time-invariant if for each (t, er, lu) in T>\ and for each \x £ T it holds that 
(r + fj,,a + /ZjW'*) is again in T>\, where lu^ is the shift of uj defined in 
equation (2.8), and 

\(t + fi, a + /i, uj^) = \(t, a, uj) 

for each t £ [<7, r). □ 

As for systems, one also defines V-completeness of behaviors, with respect to 
any family of controls (for continuous-time systems, these will be the measurable 
essentially bounded ones). 

Intuitively, complete behaviors are those with the property that the output 
is defined for every possible input, and time-invariant ones are those for which 
the same input but applied at a different time will still produce the same output. 

Exercise 2.2.6 Prove the following facts, for initialized systems: 

1. If £ is (V-)complete, then A^ also is. 

2. If S is time-invariant, then As also is. 

Give counterexamples to the converse implications. □ 

2.3 Discrete-Time 

Definition 2.3.1 A discrete-time system or i/o behavior is one for which 

T = 1. ' □ 

Assume that E is a discrete-time system. For each t £ Z, x £ X, and u £ It 
for which the right-hand side is defined, let 

V{t,x,u) := <f>(t+ l,t,x,uj), (2.10) 
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where u) is that control of length 1 in ll'*' that has the value u>(t) = u. This 
defines a next-state or transition mapping 

V :£^X, 

where 

£ = £(((,) = {(t, x, u) € Z x X x IX | (t + 1, t, x, u) S L>4 . 

In the particular case of complete systems, £ = Z x X x IX. In general, the fol- 
lowing property is an immediate consequence of the nontriviality and restriction 
axioms: 

(*) For each x € X, there exist some u € IX and some t s Z such that (£, x, u) € 
5. 

It is possible to recover the system description from the knowledge of V '. For 
instance, if oj g l(i*'' +2 ), then 

<£(t + 2, t, x, w) = V(t + 1, P(t, x, cj(t)),uj(t + 1)) 

provided that both applications of V on the right-hand side are defined, and 
4> is undefined at (t + 2,t,x,u>) otherwise. By induction one can establish the 
following fact: 

Lemma/Exercise 2.3.2 If £ is a subset of Z x X x IX that satisfies (*) and 
V : £ — > X is any mapping, then there exists a unique discrete-time system 
E = (Z, X, IX, 4>) such that £ = £{4>) and property (2.10) holds on £. □ 

The triple (X, IX, V) provides a local-in-time description of the discrete-time 
system E. One often refers to "the discrete-time system E = (Xjllj'P)" when 
in fact meaning the system induced through Lemma 2.3.2. Alternatively, one 
may introduce such a system through its associated evolution equation 

x(t + l) = V{t,x{t),uj{tj) , te Z 

together with, for systems with outputs, the measurement equation 

y(t) = h(t,x(t)). 

We often drop the ^-arguments and simply write 

x + = V(t, x,u), y = hit, x) . 

Note that if E is time-invariant then V and h are independent of t, and vice 
versa. 

In the special case in which X and IX are finite sets, discrete-time systems 
are called automata in computer science. For systems with outputs, and y also 
finite, the terminology there is sequential machines. 

In order to develop some intuition with the formalism, we analyze a typical 
sequential machine problem. Later we concentrate on systems for which all 
of the sets appearing are infinite — in fact, real or complex vector spaces — 
but automata and sequential machine theory did provide much of the original 
motivation for the development of the basic system theory concepts. 
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Example 2.3.3 In digital data communications, one is interested in reliably 
transmitting a sequence of bits (zeros and ones) over a noisy channel. In order 
to detect errors, one extra bit often is added to each block of, say, k bits. This 
parity check bit is made to be a or a 1 as needed so that the resulting sequence 
of k+ 1 elements has an even (or odd, depending on the conventions) number of 
l's. Assume that we are interested in designing a system that will detect errors. 
That is, after each k + 1 inputs, the output will be "0" if the parity is even, 
and "1," indicating that an error occurred, otherwise. While scanning a block, 
no error should be signaled. For simplicity, we take the case k = 2. (A more 
typical value is k = 7.) 

The desired i/o behavior can be modeled as follows. We chose an initial 
time, say, t = 0, and assume that the inputs will be presented starting at this 
time. Thus we chose 

u = y = {o, 1} 

and the following domain for A: 

p A = {(T,o, w )|r>o, w eU^}. 

The response map is A(t, 0,w) = 

1 if oj{t — 3) + lu(t — 2) + lo(t — 1) is odd and 3 divides r > 0, 
otherwise. 

We formalize the problem as that of finding an initialized discrete-time system 
(£,x°) with outputs whose i/o behavior matches A for r > 1, that is, 

A° S ; =A^ (2.11) 

for all t > 1. Here we will just guess a solution. One possibility is as follows: 
Consider the complete system with 





X = 


= {0,1} 3 




having the following loca. 


i description 






Xi(t + 1) 

x 2 {t+l) 
x 3 (t+l) 

y(t) 


= «(t) 

= Xl(t) 

= x 2 {t) 
= a{t).{[x 


i(t) + x 2 (t) + x 3 (t)] 


mod 2} 


and initial state x° arbitrary, where 








a:= { I 


if 3 divides t, 
otherwise. 





(2.12) 



We are using Xi to denote the ith coordinate of x G X. It is easy to see that 
this system has the desired input/output behavior. 
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The system is time- varying. Alternatively, we could also use the time-invar- 
iant system which has 

X = {0,l} 3 x{0,l,2}, 



and equations 




Xl(t + 1) 


= w(t) 


x 2 (t + l) 


= Sl(*) 


x 3 (t + l) 


= x 2 (t) 


Xi(t + 1) 


= Xi{t) + 1 (mod 3) 


y(t) 


= P(x 4 (t)).{[xi(t) + x 2 {t) + x 3 {t)} mod2} 



where /?(1) = /3(2) = 0,/3(0) = 1. As initial state we may take for instance 
(0, 0, 0, 0). One can always include a "clock" in order to pass from a given system 
to a time-invariant system that behaves as the original one. Mathematically, 
however, this associated time-invariant system is a different object; for instance, 
the set X has changed. Moreover, many desirable properties may be lost in the 
reformulation; for example, the first system that we gave is linear over the field 
GF{2) (that is, Z2 under mod-2 operations) in the sense to be defined below, 
but the corresponding time- invariant system is not. □ 

Of course, there are infinitely many different systems £ that solve the above 
problem. For instance, we change X to {0, 1} and add an equation such as 

Xi (t + 1) = 

to the equations (2.12), and this will still produce the desired behavior. Note 
that the value of X4 is not in any way observable from the output data. 

Exercise 2.3.4 The first system given above has a state set of cardinality 8, 
while for the second (time-invariant one) this cardinality is 24. Find: 

1. A system (and initial state) with state space of just two elements {0,1} 
for which (2.11) still holds. 

2. A time-invariant system with a 6-element state set for which this also 
holds. 

3. Prove that it is impossible to solve the problem using a time-invariant 
system with less than 6 states. □ 

Example 2.3.3 serves to illustrate the fact that measurement maps are useful 
not only in modeling constraints imposed by what can be observed in a physical 
system but also the fact that only certain functions of the states may be of 
interest in a given problem. 
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2.4 Linear Discrete-Time Systems 

In this section we let IK be any field, such as K, C, or the field of two elements 
GF{2) that appeared in Example 2.3.3. The terms vector space and linear will 
always be interpreted with respect to this fixed field K. 

The Cartesian product of any two vector spaces X and 11 is endowed naturally 
with a vector space structure (the direct sum) under coordinatewise operations: 

(x, u) + k(z, v) = (x + kz, u + kv) . 

Similarly for any a < r we may also consider W a ' T > as a vector space, with 
pointwise operations: 

(kw + v)(t) := ku){t) + v{t) . 

If a space is finite dimensional and a basis has been chosen, we use column 
vectors to list the coordinates; if this space is K", we always assume that the 
standard basis has been chosen, and we identify linear maps K m — * K™ with 
their matrices in the respective standard bases. When T = Z and 11 = K m , we 
identify w- a ' T ' with ll fc , where k = m(r — a), and write coordinates of elements 
in this space by listing those of lu(o~), . . . ,lo(t — 1) in that order. 

Definition 2.4.1 The discrete-time system £ is linear (over the field Kj if: 

• It is complete; 

• X and IX are vector spaces; and 

• V(t, ■, •) is linear for each t € Z . 

The system with outputs £ is linear if in addition: 

• V is a vector space; and 

• h(t, •) is linear for each t € Z . 

The system is finite dimensional if both IX and X, as well as V for a system 
with outputs, are finite dimensional; the dimension of £ is in that case the 
dimension of X. □ 

Recall that completeness means simply that V(t,x,u) is defined for every 
triple (t, x,u). Linearity is equivalent to the existence of linear maps 

A{t) : X -> X and B{t) : U -> X, teZ, 

(namely, V(t, -,0) and V(t,0, •), respectively) such that 



V(t,x,u)=A(i)x + B(i)u (2.13) 
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For systems with outputs also 



h(t,x)=C(t)x (2.14) 



for some linear 

c(t) : x^ y, teZ. 

The time- invariant case results precisely when all of these maps are independent 
oft. 

For finite dimensional systems, the integers n, m, and p are conventionally 
used for denoting the dimensions of X, IX, and y, respectively. Thus, in a basis 
each A(t) is of size n x n, B(t) is n x m, and C{t) is p x n. 

One typically specifies discrete-time linear systems simply by giving the fami- 
lies of maps {A(t)}, {B(t)}, and {C(t)}, or, in the finite dimensional case, their 
matrices in some basis. For instance, equations (2.12) correspond using the 
standard basis for GF(2) 3 to 

'0 o o s 
A{t)=\l | , B(t) = | | , C(i) = (a(t) a(t) a{t)) 
1 / 

and a is the function defined before. 

Note that, given any linear difference equation 

x(t+ 1) = A(t)x(t), 

we can think of this as a discrete-time linear system with no controls, simply 
by letting IX be the trivial vector space {0} and B(t) = 0. 

Linearity of the local- in-time transitions gives linearity of the transition map, 
by an easy induction: 

Lemma/Exercise 2.4.2 Prove that, for any linear discrete-time system and 
any pair of integers a < r, 4>(t, a, ■, ■) is linear. □ 

Linear Discrete-Time Behaviors 

We now define linear behaviors. We choose a definition that incorporates the 
fact that an input identically equal to zero should not affect the system. 

Definition 2.4.3 The discrete-time i/o behavior A is linear (over the field K.) 
if: 

• It is complete; 

• IX and y are vector spaces; 

• For every a <t in7L, X(a,a,o) = and the mapping A CT,T is linear; and 
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• For each a < fi < r in Z and eac/i input lo € IX^' 1 "-', ii holds that 

X(r,a,Ou) = \(t,h,uj), 

where is the input identically equal to on [a, /J,). □ 

A number of elementary facts hold for linear i/o behaviors and about their 
relation with linear systems initialized at x° = 0. We leave them as (easy) 
exercises. All systems and i/o behaviors are assumed to be discrete-time here. 

Lemma/Exercise 2.4.4 If £ is linear, then Aj^o is linear. □ 

Lemma/Exercise 2.4.5 If A is linear, then there exists a unique family of 
linear maps 

Aij, i,j eZ,i> j 

such that for each a < r 

T-l 

\^{lo) = Y j A t MJ) (2-15) 

for allw G U^ a ' T \ □ 

Lemma/Exercise 2.4.6 If £ and A are linear, then 

A s ,o = A 

if and only if 

2 _ f C(i)B(j) if i = j + l, r . 

y \ C(t)i4(»-l)i4(i-2)...A(j + l)B(j) otherwise l ' 

in terms of the linear maps introduced in Lemma 2.4.5. □ 

Lemma/Exercise 2.4.7 Let A be a linear i/o behavior. Consider the family 

of linear maps < Aij > introduced in Lemma 2.4.5. Then, A is time-invariant 
I J i>j 

if and only if there exists a sequence of linear maps {Ai, i = 1, 2, 3, . . .} such 
that 

•/xij — s\i— j 

for each i > j. □ 

The sequence {Ai} is called the impulse response of A (or of a system 
realizing A), since its columns result from the application of an "impulse" in 
one input coordinate, i.e., an input that is equal to zero everywhere except at a 
single instant, and at that instant the value equals 1. Another common name 
for the elements of this sequence is Markov parameters. 
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In the time-invariant case, an input lo € U^' 1 "' produces the final output 
value 

r-1 

y{r) = Y J A T -MJ)- (2-17) 

In other words, the output is the (discrete) convolution of the input with the 
impulse response. 

We conclude from Lemmas 2.4.6 and 2.4.7 that a discrete-time linear time- 
invariant system realizes a discrete-time linear time-invariant behavior precisely 
when the following factorization condition holds among their associated linear 
operators: 

' Ai = CA i ~ 1 B for all i > ol (2.18) 



2.5 Smooth Discrete-Time Systems 

In this section K = R or C. The first part of Appendix B on Jacobians and 
differentials should be consulted for terminology regarding smoothness and Ja- 
cobians. 

For any discrete-time system £ and each t € Z, let 

£ t ■= {(x,u) | (t,x,u) e £} C X x IX 

be the domain of V(t, •, •). 

Definition 2.5.1 Let k = 0, 1, 2, . . . , oo. A C k discrete-time system (over 
K.) is one that satisfies, for some nonnegative integers n,m: 

1. X is an open subset o/K"; 

2. 11 is an open subset o/K m ; and 

3. For each t € Z, the set £ t is open and the map V(t, •, •) is of class C k there. 

IfE is a system with outputs, it is required in addition that y be an open subset 
of some Euclidean space K p , and that h(t, •) be of class C . □ 

When k = 1 or oo, one also says that £ is differentiable or smooth, respec- 
tively. 

We define linearization only for real systems. In the complex case, differ- 
entiability is understood in the sense that the real and complex parts must be 
differentiable, as discussed in the Appendix on differentials, and linearizations 
could be computed for the real system of twice the dimension associated to any 
given system over C 

Definition 2.5.2 Let £ be a C 1 discrete-time system, over R, and assume that 
r = (£, u>) is a trajectory for E on an interval X. The linearization of E 
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along r is the discrete-time linear system S*[T] with local-in-time description 
,T*), where 



pn mm 



V, { t,;-):-{ V it ---- ) ' mM,)] IIH (2.19) 

7/E is a system with outputs, then S*[T] is the discrete-time linear system with 
outputs having the above V , y = R p , and the readout map 

When X = Z an<i £ and u are both constant, £(£) = x and u>(t) = u, then E*[r] 
is ifte linearization of S at (x,u). □ 

When r is clear from the context, we write simply £*. 

Note that a pair (a;, u) gives rise to a constant trajectory for E, as in the 
last part of the above definition, if and only if V(t, x, u) = x holds for all t. We 
call such a pair (x,u) € X x IX an equilibrium pair and the corresponding x 
an equilibrium state. If (x,u) is like this and E is time-invariant, then £* is 
again time-invariant. 

In terms of the matrices appearing in equations (2.13) and (2.14), the system 
E* has 

A(t) = v x (t,m,u>(t)) 

B(t) = Vu(t,£(t),Q(t)) 

whenever t G X and A(t) = B(t) = for t (JL X, and 

c(t) = h x (t,at)) 

for t £ X and C(t) = otherwise. (Subscripts indicate partial derivatives.) 
As an example take the system E: 

x + = (2x + uf 

and its equilibrium pair (x = l,u = — 1). In terms of perturbations of these 
values 

x = 1 + x, u = — 1 + u, 

the equations are 

x + = x + - 1 = (2x + u) 2 -l 



{2{l + x}-l + u) 2 -l 
4x + 2u + Ax 2 + u 2 + 4 
4x + 2u + o{x,u) . 
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Thus, E* has matrices A = (4) and B = (2). Alternatively, one could of course 
obtain these matrices by evaluation of 

V x = 4{2x + u), V u = 2{2x + u) 

at the equilibrium pair. Thus, £* is 

x + = Ax + 2w , 

which is a time-invariant system since the trajectory was a constant one (equi- 
librium pair). A variation of this example is provided by the same system £ 
but along the following trajectory onI= [0, +00): 

w(0) = 1, u(t) = for all t > 1 

(that is, uj is an impulsive input) and 

£(0) = 0, £(£) = 2 2 '~ 2 for alH > 1 , 

which results in the time- varying linearization 

x + = a t x + b t u , 

where a t = b t = for t < 0, ciq = 4, 60 = 2, and 

a t = 2 2 ' +1 , b t = 2 2 ' 

for each t > 1. 

Abusing terminology, one often simply says that (A, B), or (A, i?, C) for sys- 
tems with outputs, is the linearization of the system along the given trajectory. 
Note that for systems with no controls, that is, classical dynamical systems, 
the linearization of x + = V(x) along a trajectory is just the linear recursion 
x + = A(t)x, with A(t) being the Jacobian of V computed at each step. 

2.6 Continuous-Time 

In this Section we define continuous-time systems as those that are described 
by finite dimensional differential equations with sufficiently regular right-hand 
sides. More generally, one could also consider systems whose evolution is de- 
scribed by partial or functional differential equations, or systems with a discon- 
tinuous right-hand side, but these more general concepts will not be treated in 
the present volume. 

Motivational Discussion 

Essentially, we wish to define continuous-time systems as those that are de- 
scribed by differential equations 

x= f(t,x,u) 
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where the right-hand side may depend on t. For a well-defined system to arise, 
we need the solutions of this equation for arbitrary initial conditions and mea- 
surable locally essentially bounded controls to exist at least for small times. 
Weaker regularity hypotheses can be made, but as far as x and u are concerned, 
we shall settle essentially on continuity on both x and u and differentiability on 
x. This still leaves to be settled the issue of what regularity must be assumed in 
t, when dealing with time- varying systems. In order to decide what is reasonable 
in that regard, consider one of the main ways in which time- varying systems 
arise. Typically, there is a trajectory (£, v) that has been given for the above 
equation, and one wants to compare Q with a trajectory (£, u>) corresponding 
to a different control. In other words, one is interested in the dynamics of the 
deviations 

e := £ - C, J := co - v 

from (C,v). Note that, omitting the "£" arguments where clear, 



where 



g(t, x, u) := f{t, x + C(t),u + u{t)) - f(t, ((t),v(t)) . 



Even if / is independent of t, the function g will in general depend on t. But 
this dependence of g on t, assuming that / = f(x,u), is at most through a 
measurable essentially bounded function — v or the continuous function £ — 
substituted into /. Motivated by this, we will define systems asking that the 
time variation appear in precisely that way (substitution of a measurable func- 
tion into a continuous one). Differentiable and smooth systems will be defined 
by substitutions into differentiable or smooth functions. An advantage of do- 
ing things in this manner is that deviations such as those above remain in the 
class, and many results can be proved just for time-invariant systems, with the 
generalizations to the time-varying case being immediate. 

The Definition 

One important technical point is the introduction of concepts from Lebesgue 
integration theory. Appendix C on ordinary differential equations, which should 
be consulted before covering this section, reviews the basic notions. However, 
readers who have not seen measure theory before should still be able 
to follow the rest of this book. Most results still will be correct if one 
substitutes piecewise continuous functions instead of "measurable." 

In this section, K = R or C. At the beginning we wish to allow control 
values to belong to an arbitrary metric space 11; but for most results we later 
assume that 11 is an open subset of a Euclidean space R m or C m . The use of 
more general metric spaces permits including discrete control- value sets, which 
appear in optimal control theory as well as in other areas. 
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Definition 2.6.1 Let X be an open subset o/K" and let U be a metric space. 
A right-hand side (rhs) with respect to X and IX is a function 

/: RxlxU^K" 

which can be obtained in the following way: There must exist another metric 
space S as well as maps 

f : 5xlxli^K" 

and 

so that 

f(t,x,u) = f(*(t),x,u) (2.21) 

and the following properties hold: 

1. f(s, -,u) is of class C 1 for each fixed s, u ; 

2. both f and the partial derivative f x are continuous on S x X x IX ; 

3. -k is a measurable locally essentially bounded function. □ 

As discussed above, this definition is somewhat complicated by the need to 
deal with rather general time variations. For / independent of t, the definition 
reduces to the requirement that IX be a metric space and 

/ : X x IX -» K™ 

is so that 

f(-,u) is of class C for each fixed u (2.22) 

and 

/ and f x are continuous on X x IX . (2.23) 

Also, of course, if IX is any open subset of K m and 

/:Kxlxll^r 

is continuously differentiable, then / is a rhs. 

Lemma 2.6.2 Any rhs in the sense of Definition 2.6.1 satisfies the following 
property: For any real numbers a < t, any measurable essentially bounded 
ui G W-' T ' T ', and any x° G X, there is some nonempty subinterval J CI := [a, r] 
open relative to X and containing a, and there exists a solution of 

i(t) = f(t,S(t)Mt)) (2-24) 

S(a) = x° 

on J. Furthermore, this solution is maximal and unique: that is, if 

C : J' -> X 



44 2. Systems 

is any other solution of (2.24) denned on a subinterval J' C I, then necessarily 

J' C J and £ = C on J' . 

If J = I, then w is said to be admissible for a; . 

Proof. Consider 

g(t,x) := f{t,x,co{t)) = f{n{t),x,co{t)) 

as a function on I x 1. Note that this is continuous on x, so hypothesis (H2) 
in Appendix C is satisfied for g. Since the mapping 

I^Ux5, (m (w(t),ir(t)) 

is measurable, g(-, a;) is the composition of a measurable and a continuous func- 
tion and is therefore measurable in its first coordinate. Thus, (HI) holds too. We 
claim that the conditions in the existence and uniqueness Theorem 54 (p. 476) 
in the Appendix hold. Because of Proposition C.3.4, we need to verify that 
for each compact subset K C X and each x° £ X there exist locally bounded 
measurable maps a and (3 so that 

\g(t,x°)\<P(t) 

for all t el, and 

\g x (t,x)\ < a(t) 

for all x £ K and all t £ I. 

Since u) is bounded and n is locally bounded, there exist compact sets K\, K 2 
such that u(t) £ K\ and n(t) £ Ki for almost all t £ X. By continuity of / and 
f x , there are upper bounds M\,M<i for the values of |/| and |/ x |, respectively, 
on 

K 2 x K x Ki , 

so the functions a = M 2 and (3 = Mi are as desired. I 

Lemma/Exercise 2.6.3 Let /be a rhs, and consider 

T> := {(t, <7, x,u>) \ <t < t,x £ X,uj £ li^ IJ ' T ' is admissible for x } . 

(Note that, in particular, admissibility means that such w's are measurable.) 
On this set define 

<A(t, ct, x, w) := £(r), 

where £(r) is the (unique) solution of (2.24) on [a, r]. Then, E/ := (R, X, U, </>) 
is a system. □ 

Definition 2.6.4 ^4 continuous-time system ('o^erK = R or C) is a system 
of the type £/, where f is a rhs. □ 
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Note that this system will be time-invariant when / is independent of t. We 
often refer to Ej simply as "the system x = f(t,x,u), n and call / the "rhs of 
E." The map / is also said to provide a local in time description of E. 

Definition 2.6.5 A continuous-time system with outputs is a system E 
with outputs whose underlying system is a continuous-time system, y is a metric 
space, and for which h : T x X — > y is continuous. □ 

Definition 2.6.6 Let k = 1,2, .. . ,oo. A continuous-time system of class 

C k is one for which 11 is an open subset of K m , for some nonnegative integer 
m, and for which S and f in (2.21) can be chosen so that S is an open subset 
of a Euclidean space and f is of class C k . 

If ' E is a system with outputs, it is required in addition that y be an open 
subset of some Euclidean space K. p , and that h(t, •) be of class C k for each t. □ 

Every continuous-time system is of "class C°" since / is continuous by def- 
inition. When k = 1 or oo, one also says that E is differentiable or smooth, 
respectively. Note that, in particular, f(t, •, •) is of class C k when the system is. 

Note that a time-invariant continuous-time system E is of class C k precisely 
if U is an open subset of IK™ and / is fc-times continuously differentiable. 

It is useful sometimes to consider also the "time reversal" of a system. As- 
sume that E is a continuous-time system, and pick any /j6R. Then 



is also a rhs, since 



g{t,x,u) := -f(fjt-t,x,u) 
g(t,x,u) = g{f3(t),x,u) 



under the definition 

g : S x X x U — > K™, g(s, x, u) := —f(s, x, u) 

and j3{t) := — 7r(/i — i), where /, /, S, and 7r are as in the definition of rhs. 

Definition 2.6.7 //E and fi are as above, the continuous-time system with rhs 

-f{H-t,x,u) 

is the reversal of E at \x, and is denoted E~ . If f is independent of t, then 
E^ is time-invariant and independent of the particular [i taken, and is denoted 
just as E~. □ 

Lemma 2.6.8 Assume that o~,t G M, ct < r, let E be as above, and let <p~ 
denote the transition map of E T +T . Then, for any i,z£l, and any to G VL'-' 7 ' T ' , 

<J)(t, a, x, u>) = z iff 4>~{t, o, z, v) = x , 

where v{t) := ui(a + t — t). 
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Proof. Consider the trajectory (£,w) with £(u) = x,£(t) = z. Then, the 
function 

C(t):=Z(a + T-t) 

satisfies the differential equation with control v and rhs 

-f(a + T-t,x,u) 

and has initial state z and final state x. Uniqueness of solutions gives the desired 
conclusion. I 

Definition 2.6.9 A continuous-time behavior is one for which 

• T = K; 

• 11 is a metric space; 

• V is a metric space; 

and for each a < r it holds that the domain of each X aT is an open subset of 
£y*(a,T) and \ a ' T is continuous. □ 

For any continuous-time system E and each initial state x°, the i/s response 
map 

u> f— > 4>{t, a, x ,u>) 

will be shown to be continuous in Theorem 1 (p. 57) (see also Remark 2.8.1 for 
the non-time-invariant case) for each fixed cr, r. By composition, the following 
result then holds: 

Proposition 2.6.10 If (S,x°) is an initialized continuous-time system, then 
its behavior A is a continuous-time behavior. □ 

Remark 2.6.11 It is also true that the complete i/o map u> i— > Ae(t, cr, w) is 
continuous as a mapping into C ([cr, r], V), the space of continuous functions from 
[a, t] to y with the uniform metric. This is because the map ip that sends initial 
states and controls into the complete path £(•) is also shown to be continuous 
as a map into C ([cr, r], X) in the same Theorem. □ 

2.7 Linear Continuous-Time Systems 

In this section, K = R or C. Appendix C.4 on linear differential equations 
summarizes needed background material. 

Lemma/Exercise 2.7.1 Let m, n be nonnegative integers, X := K™, U := K m . 
Assume that 

f(t,x,u)=A(t)x + B(t)u, (2.25) 



2.7. Linear Continuous-Time Systems 47 

where A(t) is an n x n matrix and B{t) is an n x m matrix, each of whose entries 
is a locally essentially bounded (measurable) function M. — > K. Equivalently, 

/:RxlxU^l 

is linear in (a;, u) for each fixed t, and is locally bounded in t for each fixed 
(x,u). Conclude that / is a rhs. □ 

Thus, the following definition is justified: 

Definition 2.7.2 The continuous-time system £ is linear (and finite dimen- 
sional) (over K = K or <C) if it is of class C 1 and: 

• X = K n and U = K m ; and 

• Its local-in-time description satisfies that f(t, •, •) is linear for each t e R. 
The system with outputs £ is linear if in addition: 

• y = KP; and 

• /i(£, •) is linear for each t € K . 

T/ie dimension o/ S is i/ie dimension of X. □ 

Thus, there exist matrix representations as in (2.13) and (2.14). As in the 
discrete-time case, we just say "the system (A(t), B(t))" (or (A(t),B{t), C{t))) 
in order to refer to this system. By definition, we are considering only finite 
dimensional systems; to define the general infinite dimensional case would re- 
quire notions of semigroup theory. We are allowing the degenerate cases n = 
(system of dimension 0) and, more interestingly, m = 0; in the latter case, we 
interpret £ as a system with no controls and rhs f(t,x) = A{t)x. 

A linear system is by definition one that is either a discrete-time or a con- 
tinuous-time linear system. 

Lemma/Exercise 2.7.3 Prove that every linear continuous-time system is 
complete (with respect to the class of all measurable essentially bounded con- 
trols). {Hint: Use the result given in the Appendix which assumes a global 
Lipschitz condition on the rhs.) □ 

Lemma 2.7.4 Let S be a linear continuous-time system, and let a < r be in 
K. Then, 0(t, cr, •, •) is linear as a map X x £^ — * X. 

Proof. Note first that 4> is indeed defined on all of X x C^, by completeness. 
Now assume that i,z£l and uj, v € \v a,T \ and pick any r € K. Let £, £ satisfy 

i(t) = A(t)ffl + B(t)w(t) 
C(t) = A(t)((t)+B(t)v(t), 
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with £(o~) = x, C(c) = z. Then, £ + r£ satisfies a similar equation, with control 
lu + rv, and 

£(<r) + rC,{a) = x + rz . 

It follows by uniqueness of solutions that 

<P{t, a, x + rz, lu + rv) = £(t) + rC,(r) = <f>(r, a, x,ui) + r<p(r, a, z, v) 

and hence <j> is linear. I 

This result is also an immediate consequence of the variation of parameters 
formula, which here becomes 

£(£) = <S>(t,cr )x + J $(t,s)B{s)io{s)ds. (2.26) 

Exercise 2.7.5 This problem illustrates that one may be able to consider larger 
classes of controls than just bounded ones, provided that the system has a special 
structure (for instance, if it is linear). 

(a) Prove that if / is as in equation (2.25) then also arbitrary integrable inputs 
may be applied. More precisely, show that in this case equation (2.24) has 
a solution, at least for small 5 > a, for any a < t, x G K™, and any 
integrable u> € ift 7 ^. 

(b) Give an example of a (necessarily nonlinear) rhs (with U = X = R), an 
x € R, and an integrable u> G W L ' T ' T \ for some a and r, such that (2.24) 
has no possible solution on any nontrivial interval [a, 6} , no matter how 
small 5 — a is. (Hint: by definition of absolute continuity, f(t,£(t),u)(t)) 
must be integrable as a function of t if £ is a solution. Find an example 
with u(t) = i" 1/2 e U^' 11 and w(0) arbitrary.) □ 

Remark 2.7.6 The above exercise justifies the definition, for any given con- 
tinuous-time linear system E, of a "different" system T,\ for which integrable 
controls are allowed, or for that matter, say, the system E? corresponding to 
using only L 2 controls. The systems Ei and YA are basically the same as Ej, 
since <j) 1 and 4> 2 are just extensions of <fi. In such cases one uses the expression 
"E^ with integrable (or, square integrable, etc.) controls" to refer to such an 
extended system. The L 2 case is especially important, since a Hilbert space 
of controls is useful in order to understand quadratic linear optimal control 
problems. In other contexts, for instance when defining the impulse response of 
a system, often it is useful also to introduce even more general controls, such as 
distributional inputs. □ 

Linear Continuous-Time Behaviors 

Definition 2.7.7 The continuous-time i/o behavior A is linear (over the field 
K) if: 
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• It is complete (with respect to essentially bounded controls); 

• It = K m and y = KP; 

• For every a < r m K, A(ct, <t, o) = and £/ie mapping A <T,T is a linear 
(bounded) operator; and 

• For eac/i <r < /i < t m i and eac/i measurable bounded input lo € IX^' 1 "' »£ 
/io/ds that 

\(t,<t,Ouj) = \(t,h,U)), 

where is the input identically equal to on [a, /i) . □ 

Linear continuous-time behaviors can be characterized in terms of integral 
operators with respect to finitely additive measures. We do not pursue such a 
characterization here, and concentrate instead on a subclass that corresponds to 
those operators for which the complete i/o maps A are continuous with respect 
to the C 1 topology on controls. This subclass will be general enough for our 
intended applications. The next definition should be compared to the conclusion 
of Lemma 2.4.5; of course, one could just as well start with an abstract notion 
of integral operator and then make the following a conclusion, as in the dis- 
crete-time case. 

Definition 2.7.8 The linear behavior A is an integral behavior if there exists 
a measurable locally essentially bounded mapping 

K:{(t,<t) eR 2 |a<T}^K pxm 

such that 

X a ' T (uj) = / K( T ,s)u)(s)ds 

J <7 

for any a < r and any ui G \V- a ' T > . □ 

Lemma/Exercise 2.7.9 The kernel K is uniquely determined (up to a set of 
measure zero) by the behavior A. □ 

Lemma/Exercise 2.7.10 If £ is linear and A is an integral behavior, then 

A s ,o = A 
if and only if 

K(t,T) = C(t)t>(t,T)B(T) 

for almost all (t,r). Thus, the integral behavior A is realizable by some linear 
system if and only if there exist time- varying matrices 

(A(t),B(t),C(t)) 

such that the above conditions hold. □ 
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Lemma/Exercise 2.7.11 The linear integral behavior A is time-invariant if 
and only if there exists a matrix of locally bounded maps 

such that 

K(t,r) = K{t-r) 

for almost all r < t. □ 

Thus, in the time-invariant case, a bounded input uj G £^(cr, r) produces 
the final output value 

y(r)= / K(T-s)u{s)ds. (2.27) 

As with discrete-time systems, the output is a convolution of the input with the 
impulse response. 

The map K is once again called the impulse response, since just as in the 
discrete-time case it is the output that results from an impulsive input at time 
0, assuming that one interprets impulsive inputs as limits of step functions of 
unit area and support tending to zero: 

Exercise 2.7.12 Assume that K in formula (2.27) is continuous. Pick any 
fixed i G {l,...,m} and any T > 0. For each e > 0, let uo e be the control 
defined on the interval [0, T] by 

-e, if < t < £, 



1 otherwise, 

where e, = (0, . . . , 0, 1, 0, . . . , 0)' is the ith canonical basis vector. Prove that 

lim y e (T) = Ki(T), 

where y 6 {T) = \°' T (uj s ) and Ki is the ith column of K . □ 

The condition A^.o = A is equivalent in the time-invariant case to 

K(t) = Ce tA B . 
If K(-) is known to be analytic about t = 0, we can expand into a (matrix) 



power series 

oo 



»=i v '' 



and so we conclude: 



Lemma 2.7.13 The integral behavior with analytic kernel K as in (2.28) is 
realized by the linear system (A, B, C) if and only if the factorization conditions 
(2.18) hold. □ 

This means that the realization theories of (finite dimensional) linear contin- 
uous-time and discrete-time time-invariant systems reduce to the same algebraic 
problem. 
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Linearization 

The definition of linearization is entirely analogous to the one given for dis- 
crete-time systems. Again, we only consider the linearization of systems over 



Definition 2.7.14 Let E be a C 1 continuous-time system over M., and assume 
that r = (£, to) is a trajectory for E on an interval X. The linearization 
of E along T is the continuous-time linear system E*[r] with local-in-time 
description /*, where 

/.(*,-,■)- | j/f ^ (2-29) 

// E is a system with outputs, then E*[r] is i/ie continuous-time linear system 
with outputs having the above f, y = R p , and i/ie readout map 

<>•«••>:={ r««« J}"; (2.30, 

When X = M. and £ and uj are both constant, £(t) = x and u>(t) = u, then E* [r] 
is the linearization of E at (x,u). □ 

As before, when T is clear from the context, we write simply E». Now for a 
pair (a;, u) to give rise to a constant trajectory for E is equivalent to the property 

/(£, x, u) = for all t . 

We again call such a pair (x, u) G X x U an equilibrium pair and the corre- 
sponding x an equilibrium state. As in discrete-time, if (x, u) is an equilibrium 
pair, then E* is again time-invariant, provided that E was time-invariant. 

The local description of the linearization can be given in terms of partial 
derivatives of / and h with respect to x and u, just as in the discrete-time case. 
Linearization was discussed in detail in the introductory Chapter. 

As an example, consider again the controlled pendulum (with m = g = 1); 
this is a time- invariant continuous-time system, with IX = K, X = R 2 , and 

f(x,u) = [ . 

If only positions could be measured, we could model that fact by using instead 
the system with outputs having ^ = M. and h(x) = x\. 

Given any fixed trajectory (£,u>) on an interval X, the linearization becomes 
the system with matrices 
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In particular, we may consider the equilibrium pair (0, 0); then the linearization 
becomes the time-invariant system (A, B), where 

Or we may take the equilibrium pair ((7r,0)',0), and one gets the state space 
model of the linearized pendulum for which we designed controllers in the In- 
troduction. 




Figure 2.1: Stick-balancing example. 



Exercise 2.7 .15 Assume that we try to control a stick by applying a horizontal 
force at one tip (see Figure 2.1). Under simplifying assumptions, and choosing 
appropriate units and time scales, we have an equation 



for the orientation of the stick. (We are ignoring the position of the center of 
gravity, and assume that the motion occurs in a plane.) With x\ = 9,X2 = 0, 
this is a continuous-time, time- in variant system (X = R 2 ,U = R). 

(a) Give the form of the linearized system along a general trajectory. 

(b) Consider now the same example but in a horizontal plane. After simplifi- 
cation, we use as a model 



Find the linearization of this system at the equilibrium point x\ = xi = 
u = 0. 

(c) Idem about x\ = tt/2, xi = u = 0. 

(d) In what sense is the linear system that you obtained in (b) "nicer" than 
the one in (c)? Intuitively, how should this relate to the behavior of the 
original nonlinear system near the two different equilibrium pairs? □ 

Exercise 2.7.16 Consider the ("bilinear") system x = xu, where X = 11 = R. 

(a) Give the linearized system at x = 2, u = 0. 

(b) Idem along the trajectory £(£) = e*, t > 0, u) = 1. □ 
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2.8 Linearizations Compute Differentials 

For any system E, and any fixed pair of times a < r in T, we denote 

V a , T := {(x,u>) | (t,<t,x,w) eV^CXx 11 [<t ' t) , 
and for each x€lwe let 

v a , T , x : = W\(T,a,x,w) e v^} c i[[^) . 

Note that for some a,T,x, these sets may of course be empty. 

The main objective of this section is to establish conditions under which the 
mapping 

<f>(r,a, 



i 1)1 



which has T> a ,r as its domain, is continuous and/or differentiable in an appro- 
priate sense. More importantly, we prove that its derivatives are computed by 
linearizations along trajectories. This last fact underlies the mathematical im- 
portance of linearizations, and is also the basis of gradient descent techniques 
for numerical nonlinear control (cf. Section 9.4). 

To develop some intuition, consider first the case of a time-invariant dis- 
crete-time system of class C 1 having X = l n , 11 = R m , and equations 

x+ = P{x,u), (2.32) 

and for which (0, 0) is an equilibrium point, that is, P(0, 0) = 0. Assume for 
simplicity that we are only interested in trajectories of length 2, and consider 
the map 

o = 0(2,O,-,-). 

Writing elements of 

U [0,2) = U 2 = R 2rn 

as pairs (ui,v,2), the map 

a: X x IX ^X: [x, (u\, U2)) >—* P(P(x,U\),U2) 

has as its Jacobian at x = U\ = u-i = a matrix (of size n by n + 2m) which 
can be partitioned as 

(The partial Jacobians are evaluated at x = u\ = u^ = 0.) By the chain rule, 
this is the same as 

(A 2 ,AB,B) 

where 

A = P x {0,0) and B = P u (0,0) 
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in terms of the partial Jacobians of P with respect to x and u. Thus, the linear 
mapping induced by J is 



a, [0, (0, 0)] : E" +2m -» R" : ( ui | i-» J ( Mi | = A 2 z + ABu x + Bu 2 . 

(2.33) 
This mapping is the differential, or Jacobian, of a at the point x = u\ = u 2 = 0. 
Alternatively, assume that we start again with the system E, and take its 
linearization along the trajectory T = (£,w) induced by the same initial state 
x = and control uj, that is, £(0) = £(1) = w(0) = u>(l) = 0. By definition, this 
is the linear system (time- in variant, because it corresponds to an equilibrium 
pair,) with the same matrices A, B, 

x + = Ax + Bu . 

The main observation is that the 2-step "a" map for this system is precisely the 
same as the map in equation (2.33). That is, we have in informal terms, 



linearization map of transition = transition map of linearization 



To continue the intuitive discussion, consider next time-invariant continu- 
ous-time systems x = f(x,u) over K = R. Suppose given a state x° € X, an 
input ui € L^{<r, r) which is admissible for x , and denote the corresponding 
solution as x(t) = <p(t, er, x° , uj). We will show in Theorem 1 that the same input 
ui is also admissible for all initial states x near enough x , that is, the domain of 
admissibility is open. Thus, we may consider the mapping Q(x) := 0(t, cr, x, uj), 
which assigns the final state to each initial state, given this control uj, and the 
mapping is defined on some open subset of X and takes values in X. Theorem 1 
will show that this map is continuously differentiable. For purposes only of 
simplifying this preliminary discussion, we make the additional assumption that 
4>(t, cr, x, ui) is twice continuously differentiable jointly as a function of (t, x); this 
allows us to exchange orders of derivation. We now compute the Jacobian at 
X = x° of the map O. From the differential equation 

— {t,a,x,uj) = f{<f){t,cr,x,uj),uj), 
we have, taking derivatives with respect to x, 

d (d4>\ d (d<f>\ (d<f> 

(where f x indicates the Jacobian of f(x,u) with respect to x, and we omitted 
the arguments (t,a,x,ui) in order to not clutter the formula). In addition, from 
the identity <j)(cr, a, x,uj) = x we also have that 

— {a,a,x,uj) = I 
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(n x n identity matrix) for all x near x° . Therefore Q-{t, a, x°, u>) is the fun- 
damental solution associated to the linearized (or "variational" ) system along 
the pair (x, uj) with zero input perturbation, that is, the solution of the matrix 
equation 

A(t) = A(t)A(t) , A(a) = I, 

where A(t) = f x (x(t), u>(t)) for all t € [cr, r]. The Jacobian of Q(x) = c/>(t, o~,x,uj) 
evaluated at x° is then the final value of this solution: 

G*{x°)=A{t). 

An equivalent way to state this formula, which is in the form given in Theorem 1, 
is to say that the directional derivative of Q(x) at x = x° in the direction of 
an arbitrary vector Ao, that is, 0*(x°)(Ao), is given by the solution at final 
time t, A(t), of the variational equation X(t) = A(t)X(t) with initial condition 
X(a) = A . 

Moreover, if the control- value set is an open subset of K m and f(x,u) is 
differentiable in u, then the linearized system also provides a way to compute 
the Jacobian with respect to input perturbations. To be precise, take any fixed 
input /i € £??(<7, t), and consider c6(r, cr, x°,uj + h[i). If h is small enough, 
the control uj + h[i is admissible for x°, because the set of admissible controls is 
open in the topology of uniform convergence. (This is also proved in Theorem 1; 
observe that uj + hfi € ££?(<7, r) for all sufficiently small h, because there is some 
compact subset C of IX so that oj(t) € C for almost all t, and there is some h\ > 
so that the /ii-neighborhood of C is included in If.) Let us compute the partial 
derivative of <p(t, cr, x a , uj + hfi) with respect to h, evaluated at h = 0, making 
again a simplifying assumption, in this case that this map is C 2 in (t, h). Prom 

—-(t,a,x,uj + hfi) = f(<f>(t,a,x,w + h[i),w + hfi), 
at 

we get 

d (3(f>\ d (d^\ c (± „ , , (d^\ c (± „ , , 

m{dh) = dh{m) = /*(^ + M(^J+/«(^ + Mm. 

In particular, when evaluating at h = we obtain the following formula: 



d ( d(j) 
di\dh 



,-„ - 4(,) v^ 



h=a 



B(t)n 



where B(t) = f u (x(t),cu(t)) for all t G [c,r]. Also, (f)(a,a,x Q ,uj + h/J,) 
implies that ^{t, cr, x° , uj + hfi) = when t = cr. In other words, 



— (T,a,x ,ui + hfi) 



h=0 



equals the solution A(r) of the linearized equation A = A(t)X + B(t)fi with initial 
condition A(cr) = 0. 
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A stronger statement can be made if one considers the total, not merely 
directional, derivative of <J)(t, a, x° , to) with respect to u), or, more precisely, 
since we are dealing with elements of an infinite dimensional space, the Frechet 
differential (cf. Appendix B) of a = a x a aT : ui — » (J>{t, a, x , ui), seen as a map 
from an open subset of C^f (er, t) into R™. This map is shown below to be 
continuously differentiable. The differential evaluated at w, a*[w], is the linear 
map which, when applied to any perturbation fi, is computed as the solution of 
the linearized system shown above. 

In summary, also for continuous-time systems one has the principle: 
linearization map of transition = transition map of linearization. 
The main purpose of this section is to establish this fact as a general theorem. 
We provide details only for continuous-time systems. A similar result holds for 
differentiable discrete-time systems, but this is trivial to establish and will be 
left as an exercise. Also, for simplicity, we deal first with time-invariant systems. 
For such systems, properties (2.22) and (2.23) hold, and the system is of class C k 
if IX is an open subset of R m and / is fc-times continuously differentiable. The 
time-varying case is dealt with as a simple consequence of the time-invariant 
case, in Remark 2.8.1. 

From now on, E will always denote a fixed continuous-time time-invariant 
system, a < r are in R, and T := r — a. 

As in the Appendix on differential equations, we write £^(a, r) for the set 
of all essentially bounded measurable controls 

u) : [a, t] — * 11 

or just £^ , since the interval will be fixed. We think of £^ as a metric space 
under uniform convergence, that is, with the metric 

doo(^,^) := ess. sup. {d u {uj{t),v{t)) \te [&,t]} . 

When 11 = R m , we write also C^; this is the Banach space of essentially bounded 
measurable functions [a, t] — * R m endowed with the sup norm 

IMIoc :=ess.sup.{|w(t)|,te [<j,t\\. 

In this section, we use the notation |x| (rather than ||x||) for the Euclidean norm 
in the state space X C R™, or when U C R m in the control-value space U, to 
avoid confusion in arguments in which both Euclidean and function space norms 
appear. For matrices A G R fcx ' (such as Jacobians), we use also \A\ instead of 
|| A ||, for the matrix norm corresponding to Euclidean norms in R ( and R*\ 

The main result on continuity and differentiability is given below. We include 
a weak convergence statement that is useful in optimal control theory. By weak 
convergence of a sequence luj — » lu, with all ojj's as well as ui in £^{o~,t), we 
mean that 

f'T pT 

ip(s)'ojj(s) ds — * / tp(s)'u)(s) ds 
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for every integrable function ip : [a, t] — > K m ; see Chapter 10 for more discussion 
of weak convergence. 

By a system affine in controls (or control- a ffine) we mean a system over R 
for which / is affine in u € IX = K m , so that its equations take the form 



{x) + J2^g t (x). (2.34) 



x = 5o 



This includes in particular linear systems. 

Theorem 1 Let £ be a (time-invariant, continuous-time) system, and pick any 
interval [a, t] . Consider the map that gives the final state given an initial state 
and control, 

a : V a , T — > X : (aj, to) <-* £(r) , 

as well as the mapping into the entire path 

ip : T> a>T -> C° : {x,uj) i-> f , 

where £(£) = 4>{t,a,x,Lu\[ a ^))- If the system is of class C 1 and (£,w) is any 
trajectory, and if 

X e R n and fi E £™ > 

consider also the solution X : [it, t] — > K™ of the variational equation 

A(i) = /*(£(*), w(*))A(t) + /«(£(*), w(*M*), (2-35) 

wii/i initial condition \(o~) = Xq. The following conclusions then hold: 

1. The setT>(j yT is an open subset ofXxC^, and both ip and a are continuous. 

2. Take any x and any lu admissible for x, and denote £ := -ifj(x,Lu). Let 
{luj}?^i be on equibounded sequence of controls (that is, there is some fixed 
compact K C U such that LUj(t) € K for all j and almost all t G [a, r\) 
and 

lim x j = x . 

If either one of the following conditions hold: 

(i) luj — > lu as j — ► oo pointwise almost everywhere, or 
(ii) luj — > lu as j — > oo weakly and E is affine in controls, 

then £j := ip(xj,u>j) is defined for all large j and 

.lim 11^.-^=0. 



58 2. Systems 

3. IfT, is of class C 1 , then a is of class C 1 and 

a*[x,w](\ ,n) = A(t) 

when X is as in (2.35). That is, a*[cc,w] is the same as the map a* 
corresponding to the linearization X* [£, ui\ . 

In particular, for systems of class C , 

a(x, ■) : X> CT ,r,x — > X 

has full rank at to if and only if the linear map 

C^^X-.fi^ [ ®{T,s)B(s)n(s)ds 



(the map a(0, •) for the linearization S*[r] along T = (£,w), seen as a iime- 
varying linear system) is onto. 

Proof. First note that conclusion 1 follows from statement 2(i); this is be- 
cause uniform convergence u)j — > oj in ^^(ct, r) implies in particular pointwise 
convergence, as well as — cf. Remark C.1.3 in the Appendix on ODE's — equi- 
boundedness of the sequence {luj}. We will show how the continuity statements 
2(i) and 2(ii) are both easy consequences of Theorem 55 (p. 486) found in that 
Appendix. Assume given any (x,u>) G T> a _ T and any sequence uij — » u converg- 
ing in either of the two senses, and let £ = ip(x,u>). Pick any e > 0. We wish to 
show that ||£j — £|| < £ for all large j. 

We start by choosing an open subset Xo of X whose closure is a compact 
subset of X and such that £(£) G Xo for all t G X := [a, r]. Let if C U be a 
compact set so that u)j(t) G K and u>(t) G K for almost all t€l. Introduce the 
functions X x X — > R n given by 

hjitiz) ■= f( z ,Uj{t)) - f(z,w(t)) 

for each j = 1 , 2 , . . . , and 

f(t,z) := /(*,«(*)). 

Note that all these functions satisfy the hypotheses of the existence Theorem 54 
(namely (HI), (H2), and the local Lipschitz and integrability properties) found 
in the Appendix on ODE's. For each j let gj := / + hj, so gj(t, z) = f(z, uij(t)). 
Since the Jacobian ||/ x || is bounded on the compact set closXo x K, there is a 
constant a > so that 

\\9j(t,xi) -g j {t,x 2 )\\ < a\\x 1 -x 2 \\ 

for all t G X, all j, and all X\, xi G Xo. Let 

Hj(t) := / hj(s,£(s))ds , teX 

J a 
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and Hj := sup teI \\Hj(t)\\. 

Assume for now that we have already proved the following fact: 

lim H_- = (2.36) 

(which amounts to saying that the functions Hj converge uniformly to zero). 
We wish to apply Theorem 55 (p. 486), with "X" there being X , "/" being /, 
a(t) = a, and z° and h equal to Xj and hj respectively, for all large enough j. 
Take any < D < e so that 

{z\ \\z-£{t)\\ < D for some t e [<t,t}} 

is included in Xo, and choose an integer jo so that, for each j > jo, \\x — Xj\\ 
and H_j are both less than 

£ e -o(r-<T) < ^_ e -a(r-a) 

2 ~ 2 

For any such j, using z° = Xj and h = hj in Theorem 55 (p. 486) implies that 

lle-^IL < {Wx-XjW+Hj) < e 

as desired. 

We now show (2.36). Suppose first that 0Jj(t) — > ui{t) a.e. as j — * oo. Then 
/ij(i, ^(i)) — * a.e. as well. Moreover, letting c be an upper bound on the values 
of ll/H on the compact set closXo x K, 

\\hj(t,at))\\ < 2c 

for all t € X. The Lebesgue Dominated Convergence Theorem ensures that 

/ \\hj(s,as))\\ ds -> 0, 

from which (2.36) follows. 

Suppose now that E is affine in controls and convergence is in the weak 
sense. We may write 

hj(s,$(s)) = G(e(*))k-W -«(*)]. 

where G(2:) is the matrix whose columns are g\ (z) , . . . , g m (z) . Thus the ith 
coordinate of hj(s, C( s )) ; i = I, ■ ■ ■ ,n, has the inner product form 

<Pi(s)' [wj(s) - u)(s)\ 

where fi(s), the transpose of the fth row of G(^(s)), is a continuous and hence 
integrable function of s. By the weak convergence assumption, 

ifi(s)' [wj(s) — uj(s)]ds — * as j — * oo 
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for each fixed t £l and each i = 1, . . . , n. Moreover, the functions Hj form an 
equicontinuous family, because \\Hj(ti) — Hj(t2)\\ < 2c\t\ — £2! (recall that the 
norm of hj is upper bounded by 2c). Pointwise convergence of an equicontinuous 
function implies uniform convergence, so also in this case we have proved (2.36). 
We next prove differentiability. (The proof is organized in such a manner as 
to make a generalization in Section 2.9 very simple.) Let (x,U>) be given, and 
f := 4>(x,cu). Note that now X = R™ and IX = E m . We first multiply / by a 
smooth function K n x K m — * K which is equal to one in a neighborhood of the 
set of values (£(£), w(t)) and has compact support. Since differentiability is a 
local property, it is sufficient to prove the result for the new / obtained in this 
manner. Thus from now on we assume that / is bounded and globally Lipschitz: 

\f(x,u)-f(z,v)\<M(\x-z\ + \u-v\) 

for all x, z in 1™ and all u, v in R m . In particular, solutions are globally defined. 
As a consequence of the Mean Value Theorem, we can write 

f(x + a,u + b) - f(x,u) - f x (x,u)a- f u (x,u)b = N{x 1 u 1 a,b) (2.37) 

for all x, a G K™ and u, b G K m , where N(x, u, a, b) = 



I 



l 

[(fx(x + ta,u + tb) - f x {x,u))a + (f u (x + ta,u + tb) - f u (x,u))b] dt 



(2.38) 
(see, for instance, the proof of [264], Chapter V.4, Corollary 2). In particular, 
N is jointly continuous in its arguments and vanishes when a = b = 0. Observe 
that, because N has compact support, it is uniformly continuous; thus, the 
supremum C(a, b) over all x G K", u G R m of 

l 

\f x (x + ta,u+tb) - f x (x,u)\ + \f u (x + ta,u + tb) - f u (x,u)\ dt (2.39) 
o 

is continuous and it vanishes at a, b = 0. 

Along the given trajectory (£,o>), consider the linearization matrices 

A(t) = f x (£(t)Mt)) 
B(t) = A.U(*),w(t)). 

For any other control v G C^{a, r) sufficiently near uj, and each z near x, we 
consider the corresponding trajectory (£, v), where £ = ip(z, v). Introduce 

s(t) ■.= at) - m 

and 

H(t) := v(t)-u(t). 

From (2.37) and (2.39) we conclude that 

6(t) = A(t)S(t) + B(t)n(t) + 7 (t) 
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with 

|7(t)l < (\8(t)\ + \n(t)\)C(8(t),n(t)). 

Thus, if A solves (2.35) with 

A(er) = 5(a) = Xq = x — z , 

it follows that 

5(t) = X(t)+ $(r, 5)7(5) ds, 

J <7 

where $ is the fundamental solution associated to A(-). So 

\5(t)-X(t)\ < ciOT^ + HmIIoo) ™p C(8(t),n(t)) 

te[a,r] 

for some constant C\ (where "sup" means "essential sup" ) . 

By a Bellman-Gronwall argument as in the proof of continuity, one has an 
estimate 

IWloo < C2(|Ao| + HU, 

so we can conclude finally an estimate of the type 

\5(t)-X(t)\ < c(|Ao| + |H|J sup C(5(t),f,(t)) 

te[a,r] 

for some constant c. Since the supremum is small for z near x and v near u>, 
this establishes differentiability. 

Finally, we must prove that the map a is continuously differentiable. Let 
(x, uj) be in the domain 2? CTjT . We must prove that, for each e > there is some 
5 > 0, such that for each (x, ui) G T> n ^ T for which 

||w-o5|| + |a;-£| <8, (2.40) 

necessarily 

|| a* [x, oj] — a*[x, w\\\ < e . 

By definition of the operator norm, and using the characterization of the deriva- 
tive just obtained, this is the same as asking that for each e there be a 5 such 
that, for each A G 1" and each fj, e M m , 

($(r,a)-$(r,a))Ao + J (®(t,s)B(s) - $(t, 5)^(5)) /*(*) ds 

< e(|Ao| + ||A*||) (2-41) 

when (2.40) holds, where Q(t,s) is the solution at time s of the matrix initial 
value problem 

X(t) = f x (Z(t)^(t))X(t), X(t) = I (2.42) 

and similarly with <!>, and where 

B(t) := U(S(t)Mt)) 
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and similarly for B. 

Note that the left-hand side of (2.41) is always bounded by 



|$(T,a)-$(r,cr)||Ao| + / \Q(a)\\n(a)\da, (2.43) 

where 

Q{s) := $(r, 5)5(5) - $(r, s)B(s) , (2.44) 

and also that |Q(s)| is bounded by 

\*(t,8)\\B(8)-B(8)\ + c\$(t,8)-*(t,s)\, (2.45) 

where c is any constant bounding the values 

fu{x,u) 

whenever a; is in a neighborhood of the set 

{£(*),*€ for]} 

and u is in a neighborhood of the set 

{u(t),t€ [<t,t]}. 

This implies that (the essential supremum of) |Q(s)| is small when (x, w) is near 
(x, o3): Indeed, |$(r,s) — $(t, s)| is small (consider the variational equation 
seen as a control system, and apply the continuity part of the proof), and 
\B(s) — B(s)\ is small, too, since it is obtained as an evaluation of a continuous 
matrix function along nearby trajectories. This completes the proof. I 

The reader may wish to consult Corollary 9.1.1 for a (weaker) version of this 
theorem which is stated in terms of directional derivatives, in the style of the 
discussion that preceded the proof. Also, later, in Section 4.2 (page 147), we 
establish several additional properties regarding joint continuity and differen- 
tiability of <j>(t, a, x, lo) on (t, x). It should be noted that, in the last part of the 
theorem, we actually established a more general fact on differentiability than 
stated, namely, that the map tp is differentiable. 

Remark 2.8.1 The results can be generalized to time- varying systems in a 
straightforward manner. Recalling that the definition of rhs involved the auxil- 
iary function /, it is only necessary to apply the Theorem to the system 

x = f(n,x,u) 

with 7r thought of as a new control. Fixing the value of 7r at that of the given 
function that provides the time variation, results on differentials of the original 
dynamics provide results for x = f(t,X,u). For instance, the mappings a and 
ip are still continuous, and are differentiable in the case of systems of class C 1 
with respect to the uniform convergence norm. □ 
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A very useful Corollary is that piecewise constant inputs approximate the 
effect of arbitrary controls. By a piecewise constant control 

u G U [a ' T) 

we mean a control for which there exists an integer I > 0, a sequence of times 

a = ti, . . . ,ti = t , 

and a sequence of values 

Ui,...,Ul 

in U so that 

U)(t) = u % if t G [U,ti+i) 

for each i = 1 ,...,/ — 1. 

Lemma 2.8.2 Assume that £ is a time-invariant continuous-time system and 
that IX is separable. (This is the case, for instance, if 11 is a subset of a Euclidean 
space, with the induced metric.) Suppose that 

4>(t, a,x,u>) = z 

for some x,z G X and lo G li'-' T,T '. Then there exists a sequence of piecewise 
constant controls 

Wl,U>2, ••■ ,Wfc,... 

so that (x,loj) G 2? CTiT for all j and 

Zj := <P(t,o~,x,ujj) — > z as i — > oo . 
Further, there exists a sequence of states {xj} so that (iej,u>j) G 2?<t, t for all j, 

4>{t,o,Xj,ujj) = z for all j , 

and 

Xj — > x as £ — > oo . 

Proof. The first conclusion is immediate from Remark C.1.2 in Appendix C, 
plus Part 2(i) of Theorem 1. The second conclusion follows from the same 
argument applied to the time-reversed system 

x = -f{x,u) 

(cf. Lemma 2.6.8). I 

The following discrete-time result is easy to prove by induction on the length 
of inputs: 
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Lemma/Exercise 2.8.3 Let E be a time-invariant discrete-time system of 
class C , and denote 

V atT := {(x,w) | (t,<t,x,u) eV}CXx U [ff ' r) . 

Then the following conclusions hold, when lv-' T ' T > is identified with an open 
subset of a Euclidean space K m ( T_<T ). 

As a subset of X x U\ a ' T ', the set 2? CT ,r is open, and the mapping 

a : 2?o-, T — > X, a{x,uj) := cj>(T,a,X,u>) 

is continuous. If the system is of class C , then a is differentiable; moreover, 

a*[aj,w](A ,/x) 

is the solution A(t) of the variational equation 

A(t+l) = /xK(t),w(t))A(t)+/ u K(t),w(t))M(t), t€[«7,T) (2.46) 

with initial condition A(cr) = Ao, where f x and /„ denote Jacobians of / with 
respect to the first n variables and the last m variables, respectively, and where 
X(t) s K™ and /j,(t) G R m for each t. Here £ denotes the path tp(x,u>). 
In particular, 

a(x, •) : 2?<r,T,a; ~ * 3C 

has full rank at w if and only if the linear map a(0, •) for the linearization E*[r] 
along r = (^,w), seen as a time- varying linear system, is onto. □ 

2.9 More on Differentiability* 

When II C IR m , we may consider for each p > 1 and each a < r the space 

of p-integrable functions on [cr, r]. Since the interval of definition is finite, this 

contains £^?. We shall be interested in the latter space, endowed with the norm 

of the first: 

B p 

will denote C^ with the norm 



\\u\\ p :={f\u>{t)\*d^ 



for each p G [l,oo). For simplicity in statements, we also let B^ be the same 
as Cm (with the sup norm). The normed spaces B^, p < oo, are not Banach, 
being dense subspaces of the respective Banach space of p-integrable maps. 
It is a standard real- variables fact that for each pair p, q so that 



1 < p < q < oo 

* This section can be skipped with no loss of continuity. 
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there exists a constant C\ so that 

IMI P <ci|M| g 

for all U) £ CJ^. In fact, as a consequence of Holder's inequality this holds with 

ci = (t - <r) 



1_ 1 

I V 1 



(see, for instance, [190], Theorem 13.17). Conversely, ifl < p < q < oo and if & 
is any given constant, then there is another constant c<i such that 

IK< C2 ik 

whenever ||w|| < k. This is proved by writing 

ww = wt)r p Mt)\ p <k«- p wt)\ p 

and integrating. In particular, all of the p-topologies (p < oo) are equivalent on 
bounded subsets of L^ (but not for p = oo). 

The results on continuity and differentiability will depend on the precise 
value of p. To understand why, it is worth thinking about the most simple case 
of a system with equation 

x = g(u) 

and X = IX = R. Here 



0(t,M,w)= / g(w(t))dt. 

•J a 

If u> n converges uniformly to u) and g is continuous, then also 

<t>(T,a,0,w n )-Kl>(T,a,0,u). (2.47) 

But if the convergence u>„ — » to is weaker, then (2.47) will in general fail to 
follow, unless either one knows that the convergence is dominated or if some 
other condition on g, such as linearity on u, is satisfied. 

We introduce a technical condition that will be needed later. 

Definition 2.9.1 Let the system E be of class C 1 . We will say that E has 
linear growth on controls if U is a convex subset of W n and it holds that, 
for each compact subset K C X, there is some constant k so that the rhs of E 
satisfies 

||/„(a:,«)||<fc (2.48) 

whenever x € K and u € U. If in addition E is of class C 2 and it also holds 
that the second partial derivative 

f xu {x,u) (2.49) 

is bounded in norm uniformly on K xll for each compact subset K C X, meaning 
that the second mixed partials of f with respect to all variables (xi,Uj) are all 
bounded, then we will say that E has strongly linear growth in controls. □ 
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There are two main cases of interest in control theory in which both of these 
properties are satisfied. The first is that in which the system is of class C 2 and 
the control value set IX is (convex and) bounded and / extends as a C 2 function 
in a neighborhood of closll C M m . The second is that of systems affine in 
controls as described in Equation (2.34). 

Lemma 2.9.2 Assume that E is of linear growth in controls, X = W l , and 
there exists some compact subset K\ of X such that f(x, u) = for all x ^ K\. 
Let K 2 be any compact subset of IX. Then, there exists a constant M such that 

\f(x°,u) - f(z°,v)\ < M (\x° - z°\ + \u- v\) (2.50) 

for every x° , z° G X, u G K 2 , and v € IX. Further, if E is of class C 2 and 
pg [1,2], then denoting 

g(x,/3) := f(x,0) - f(x n ,u) - f x (x°,u)(x - x n ) - f u (x°,u)(/3 - u) 

there exists a constant N so that 

\g(z°,v)\ <N(\x°-z°\ 2 +\u-v\ p ) (2.51) 

for every x Q , z° G X, u € K 2 , and v G IX. 

Proof. Property (2.50) is obtained by separately bounding 

\f{x\u)-f{z\u)\ 

and 

\f(z°,u)-f(z°,v)\ 

using property (2.48) and the Mean Value Theorem (convexity of 11 and the 
compact support property are used here). We now prove property (2.51). As- 
sume first that \u — v\ < 1. Then v is in the compact set 

K 3 :={v\\v- (3\<1 for some /3 G K 2 } . 

By Taylor's formula with remainder (recall that / is twice differentiable) , we 
know that, for all x°, z°, and u,v G K, 



3, 



2 



\g(z°,v)\ <a|a; -z | 2 + 6|w-i;| 

for some constants a, b (which depend on K\ and K 3 and hence also depend on 
K 2 ). Since \u — v\ < 1 and p < 2, this means that also 

\g(z°,v)\ <a\x°-z°\ 2 + b\u-v\ p . (2.52) 

Next note that gp(x, 0) = f u (x, (3) — f u (x° , u), so that the system having g as a 
rhs also has linear growth in controls (in equation (2.48), for the same compact 
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one must use 2fc instead of k). We can then apply the first conclusion, (2.50), 
to this other system to conclude, for z°,u,v as in the statement, that 

\g(z ,v)-g(z°,u)\<M'\u-v\ 

for a constant M' that depends only on K\ and K 2 . If \u — v\ > 1, then also 

\u — v\ < \u — v\ p ; thus, 

\g(z°,v)\ < \g(z ,u)\ + \g(z°,v)-g(z ,u)\ 

< a\x° -z°\ 2 + M'\u-v\ p . 

Choosing N := max{a,6, M'}, the result follows. I 

We now provide a p-norm version of Theorem 1 (p. 57). 

Proposition 2.9.3 Let the notations be as in the statement of Theorem 1. 
Pick any p G [l,oo). Then: 

1. If S has linear growth on controls, then the set T> a ,r is open in X x B^ and 
both ip and a are continuous (with respect to the p-topology on controls). 

2. Take any x and any uj admissible for x, and denote £ := ij){x,w). Assume 
that 

lim Xj = x and lim LUj = lu , 

j — >oo j — >oo 

where the sequence {ujj} is equibounded and the convergence is now in 
Bf n . Then £j := ip(xj,U)j) is defined for all large j and 

i™ HO-eiL = o. 

3. If p > 1 and £ has strongly linear growth on controls, then the same 
differentiability conclusion as in Theorem 1, Part 3, holds with respect to 
the p-topology. 

Proof. Consider first the continuity statements. We assume first that X = W l 
and that / is globally Lipschitz in the following sense: For each compact subset 
Ki of 11, there is a constant M such that 

|/(a;,u)-/(«,tOI<Af(|a;-*| + |u-t;|) 

for all x, z in K™, u in K 2 , and v in U. So solutions are globally defined. Assume 
that (x,u>) gives rise to the trajectory (£,w); we wish to prove continuity at 
(x,u>). Let (£, v) be any other trajectory. We have that, for each a <t < r, 

£(*)- C(*) = £(*)-<(*) + I (/«(*).«(*))- /(CW, "(*))) d«- 
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By the Lipschitz condition, using for K% any compact containing all values of 
U), there is an M so that 



\m - C(*)l < KM - <»l +M\\w-v\\ 1 + M \£(s) - ((s)\ds . 

J a 

By the Bellman- Gronwall Lemma, we conclude that 

m - cwi < m*) - cwi + m || W - nil) e Mt 

for all cr < t < r. Thus, for each 1 < p < 00 there are constants a, b such that 

|(/>(i, a, x,w) - <j>{t,o,z,v)\ < a\x — z\ + b \\w — v\\ (2.53) 

for all x, Z in K™, all v in B^, and all a < t < r. This shows the continuity of 
i/> and a at (x,lj) under the above global Lipschitz assumption. 

Assume now that X, IX, and / are arbitrary. Pick any element (x,u>) in £> ct ,t- 
Choose open subsets V and W of X such that 

V C clos (V) C W 

and so that 

£(i) G V for all t G [a, r] 

and W has a compact closure. Let 9 : K n — > K be any smooth function that is 
identically 1 on clos (V) and vanishes outside W. 

Consider now the system obtained with X' = R n , same II, and / replaced 
by h, where 

h(x,u) := 9(x)f(x,u) 

on X x U and /i = if x <£ X. If / has linear growth in u then h also does, and 
hence since 9 has compact support we are in the situation of (2.50) in Lemma 
2.9.2. Thus, h is globally Lipschitz in the above sense, and we can apply the 
first case ofj;he proof to this h. 

We let <f) be the transition map <j) for this new system. By (2.53), there is 
then a neighborhood V' of x and an e > such that <p{t, a, z, v) is in V for all 
a < t < t whenever z G V and \\u> — v\\ < e. 

The maps h and / coincide in a neighborhood of the points of the form 
(£(t),ui(t)), t G [a, r], because 9 is independent of u. Therefore 4>(t,a, z,v) 
solves the original differential equation, i.e. it equals <p(t, <r, z, v) for these t, z, v. 
In particular, T>a^ T contains a neighborhood of (x,u>) and is therefore open. 
Continuity of ip and a then follows from (2.53). 

For the dominated convergence statement, one may apply the previous case. 
The hypothesis of linear growth can be assumed, because we can always modify 
the rhs of the system in such a manner that its values remain the same in a 
neighborhood of the values of the corresponding trajectories but so that the 
growth condition is satisfied. For instance, one may multiply / by a function 
9(u) which vanishes for |u| > C. 
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We now prove differentiability, assuming that the system is of strong linear 
growth in controls. Since differentiability is a local property, we may again 
modify / (by multiplication by a scalar function) and assume that X = SJ 1 and 
/ is globally Lipschitz in the previous sense. 

As in the proof for p = oo, let (£,w) be any fixed trajectory, and consider 
the linearization matrices A(t), B(t) along it. For any other control v (of length 
T) sufficiently near uj in B^, and each z near x, we consider the corresponding 
trajectory (C,^), where Q = -ifj(z,i>). By continuity, we may choose a neigh- 
borhood of lu (in the p-norm) so that this trajectory stays always in a given 
compact convex neighborhood of x in X, say K\. Let K 2 be any compact set 
such that the (essentially bounded) control lu satisfies ui(t) G K 2 for almost all 
t. As before, introduce 5(t) := ((t) — £(£) and /x(£) := v(t) — co(t). From the 
conclusion (2.51) in Lemma 2.9.2, it follows that, if 1 < p < 2, then 

6(t) = A(t)6(t) + B(t)n(t)+7(t), 

where 

h(t)\<N(\S(t)\ 2 +\n(t)\v) 

for a suitable constant N. So if A solves (2.35) with 

X(a) = 5(a) = Xq = x — z , 
one has again the explicit form 



5(t) = X{t)+ $(t, s)t(s) ds . 

So 

\S(r)-X(r)\<M^\\l + £\5(t)\ 2 d^ 

for some constant M. Applying (2.53) to the second term there results that 
there is a constant M' such that 

|5(t)-A(t)|<M'(|H|£ + |A | 2 +|H£) . (2.54) 

Since p > 1, it follows that \8(t) — A(r)| is indeed 

*(|Ao| + IHI p ) 

as required to establish differentiability. If, instead, 2 < p, we consider equation 
(2.54) for the case p = 2. Since 

IMI 2 <c|H| p 

for some constant, it follows that \S(t) — A(r)| is majorized by an expression 

M"(|A | 2 + 2|| M ||3 , 
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again as desired. 

The proof of continuity of the derivative is as follows: As in the proof of 
Theorem 1, for each (x,u>) in the domain 2? CTjT and each e > we need to show 
that there is some S > such that, for each (x, u>) G T> atT so that 



necessarily 



<5. 



a* x, will < e . 



(2.55) 



As in the proof for the sup norm, this means that for each e there should be 
some 5 such that, for each Ao € R™ and each /i e K m , Equation (2.41) must 
hold, except that the term \\/j,\\ now should be \\fi\\ . We again have the bound 
(2.43), with Q as in (2.44), as well as the bound (2.45), with c now being valid 
for arbitrary u. 

Bounding Q and applying Holder's inequality, we conclude from (2.44) and 
(2.45) that the left-hand side of (2.43) is bounded by 



fc|A | + fc' B{-)-B{.) M\ p + {T-a)^ckM\ p 



(2.56) 



where 



k := sup |$(t, s) - $(t, s) 

s£[<t,t] 

k' := sup |#(r,s)|, 

s€[u,t] 



and q - 



v 
p-i- 



Note that k! is finite because <& is continuous in s; thus, the proof will be 



B(-)-B(.) 



and k are small for (x, u>) near 



completed if we show that both 

(x,w). 

Consider equation (2.42). This can be thought of as a system in dimension 
n 2 whose controls are £, to. Note that / xu is bounded, because we multiplied / 
by a function of x with compact support, and the original system has strong 
linear growth. In particular, then, this n 2 -dimensional system has itself linear 
growth. It follows from Part 1 that k in equation (2.56) is indeed small provided 

that 1 1 a; — a3|| and £ — £ be small, but the second of these tends to zero as 

the first does and |x — x| — » 0, by continuity. 

Finally, we need to establish that if \x n — x t 
then, with B n {t) = fu(€(i),u(i)), also \\B n {-) - B{-)\\ q -» 0. Since under the 
assumptions also ||£„ — £11^ — > and therefore ||£„ — ^|| — » 0, where ^„ = 
ip(x n ,uj n ), and the values of all £„ may be assumed to stay in a compact subset 
of the state space, it will be enough to prove the following fact (for each entry 
b of the matrix B): If 
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is a bounded continuous function and if 



then also 



K-*ip-> °; 



\bov n -bo v\\ q — > 0. 



Note that \\u) n — u>\\ — > implies that w n converges to u; in measure, that is, 
lim meas{£ € [a, r] | |w„(t) — u>(£)| > <5} — > 

n — >oo 

for each 5 > (see, for instance, Exercise 13.33 in [190]). Since b is uniformly 
continuous, also bout n converges in measure to bouo. Because the sequence {boto n } 
is uniformly bounded, it follows from the Dominated Convergence Theorem and 
the Riesz Theorem (see, e.g., Exercise 12.57 in [190]) that the convergence is 
also in (/-norm, as desired. I 

Remark 2.9.4 The differentiability result is false if p = 1. For instance, let 
a = 0, r = 1, and consider the system 

x = u 

with X = M. and IX = (—1,1). We claim that the map 

/3:= a(0, •) = 0(1,0,0,-) : v -> / v{t) 2 dt 

Ja 

is not differentiable at lu = 0, from which it follows that neither is a differentiable 
at (0,0). To see this by contradiction, write 

L:=j9*[0] 

and note that, because (3(f) = /3(— v) and Lv = —L{—v) for each v, 

2 H/3MH < ||/3(z,) - Li/ll + \\P{-v) - L{-v)\\ . (2.57) 

In particular, consider the family of controls u> £ : 

e { otherwise 

and note that \\lo £ — lo^ — > when e — » 0. Thus, it should hold that each term 
in the right-hand side of (2.57), and hence also /3(w e ), is of order 

odl^ll,) = o(e). 

Since (3(u> £ ) = e, this is a contradiction. 

Note that, on the other hand, for p > 1 one has for this example that 

IM P = £ 1/P , 

and there is no contradiction in that case. □ 
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Remark 2.9.5 For p < oo either the linear growth condition or the dominated 
convergence assumption is essential. Otherwise, not even continuity holds. This 
is because convergence in the p-topology does not imply convergence in the q- 
topology, for q > p. Indeed, take any finite p and consider the equation x = u q , 
where q > p is arbitrary. Pick any r with q > (1/r) > p. The control lu £ defined 
now by 

oj £ (t) := e~ r for t < s 

and zero otherwise, has ||w £ || — > as e — > 0, but 

for the corresponding solution £ e . □ 

2.10 Sampling 

In Section 1.3 we discussed the motivation behind the process of sampling and 
digital control. We now define this precisely. 

Definition 2.10.1 Let's = (R, X, It, </>) be a continuous-time system, and pick 
any real number 6 > 0. The <5-sampled system associated to £ is the dis- 
crete-time system Em defined as follows. Let E$ be the subset of 

Z x X x It 

consisting of those triples (k,x,u) that satisfy that lu is admissible for x, where 

is defined by 

uj{t) = u for all t e [kS, (k + 1)6) . 

The state space o/Sui is then the set of all x G X for which there is some k,cu 
with (k,x,u>) G £5, and the local-in-time dynamics are given on £$ by 

V(k,x,u) := <f>{{k + 1)6, k6, x, u>) . 

For systems with outputs, Eui has the same y and hasY,. □ 

Note that, in this definition, the effect of the control u on the sampled 
system is undefined if the solution to the differential equation does not exist for 
the entire interval [k6, (k + 1)6}. We think of the sampled system as providing 
a "snapshot" view of the original system every 6 seconds, with constant control 
applied in between; if the original system had an explosion time during the 
interval, the state at the end of the interval would be undefined. 

It is easy to show that if E is linear (respectively, time-invariant), then 
Erji is also, for any 6. For time-invariant systems, V(x,u) is just the result of 
integrating £ = f(^,u) on [0,(5], with initial state £(0) = x and control u = u. 
(And V(x,u) is undefined if the solution does not exist.) 
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Exercise 2.10.2 Find Em for the following E and 5=1 (all examples have 

X = U = K): 

(a) x = xu; 

(b) x = ax + bu (a, b are constants); and 

(c) x = (x + u) 2 (be sure to specify the state space and the domain of V for 
this example). □ 

2.11 Volterra Expansions* 

For continuous-time systems, certain representations of input to state and re- 
sponse maps are particularly useful. Before describing one of these, we review 
the Peano-Baker formula and a close relative. 

Recall that a fundamental solution of a linear differential equation $(£,<t) 
admits a power series expansion, the Peano-Baker formula (C.27) discussed 
in Appendix C.4. Substituting the power series (C.27) into the variation of 
parameters formula gives an expression for cj>(T,a,x,u>) in terms of a series of 
integrals, for linear systems, as shown in Exercise C.4.2 in that Appendix. We 
now describe an alternative such series, assuming for simplicity that x = 0. 
Replace 

i(t) = A(t)£(t) + B(t)u>(t) (2.58) 

£(a) = 

by the corresponding integral equation 

£(£)= f B{s)cj{s)ds+ f A{s)Z{s)ds, 



and solve this by iteration. There results a series representation for the solution 
as follows: 

/■t r-t r-Si 

i{t) = I B{s 1 )uj{s 1 ) dsi + A(si)B(s 2 )co{s2)ds 2 ds 1 

J a J <j J a 

/ / A(s 1 )A(s 2 )B(s 3 )uj(s 3 )ds 3 ds 2 ds 1 +... 

>a J a J (j 
ft rsi rsi-i 

/ • • • / A( Sl )A(s 2 ) . . . ^(s ; _i)B(s ; )^(s;) dsi... ds 2 dsi 
+ .... (2.59) 

Note that this series is slightly different from that given in Exercise C.4.2: the 
bottom limits of integration are now all equal to a . 



* This section can be skipped with no loss of continuity. 
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Lemma/Exercise 2.11.1 Prove that both the series (2.59) and the series ob- 
tained by taking termwise derivatives converge uniformly for t on bounded in- 
tervals, and that therefore the series (2.59) provides a solution to the initial 
value problem (2.58). □ 

Using the above we now show, for a simple type of nonlinear system, how to 
obtain an integral representation similar to the variation of parameters formula. 
The resulting representation is called a Volterra Series. 

A bilinear continuous-time system is one with X = K n , IX = K m , and equa- 
tions 

i(t)= U(i) + f^Wi(t)Ei(t))t(t) + B(t)w(t) , (2.60) 

where E\,..., E m , as well as A and B, are matrices of locally essentially bounded 
functions. Note that linear systems appear as the special case when all E^ vanish. 
A bilinear system with outputs is one with a linear measurement map 

y(t) = C(t)£(t) (2.61) 

and y = R p . (One could equally well define complex-valued bilinear systems, of 
course.) 

For notational simplicity, we restrict attention to single-input bilinear sys- 
tems, those for which IX = R and 

i(t) = A(t)S(t) + w(t)E(t)£(t) + B{t)uj{t) (2.62) 

(here B is a column vector), but the multi- input case is entirely analogous. 

We will find a series representation for solutions of this equation that exhibits 
the dependence on powers of the control u>. To start, we consider the solution 
£ of (2.62) having initial condition £(c) = 0. Introduce the function 

C(t) := t>(a,t)at) , 

where $ is the fundamental solution associated to A(t). Then, £ satisfies the 
differential equation 

C = ujF(+Glu, (2.63) 

where we introduced 

G{t) :=®(<j,t)B(t) 

and 

F(t):=*(a,t)E(t)$(t,<r). 

Using the expression (2.59), with ujF in place of A and G in the place of B, in 
order to give a series representation for the solution of (2.63), we obtain that 
£(£) equals 

t l-t I-Sx 

G(si)u>(si) dsi + / / F(si)G(s2)to(si)u>(s 2 )ds2dsi 
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+ / F(s 1 )F(s2)G{s 3 )u)(s 1 )u(s2)uj(s 3 )ds 3 ds2ds 1 



F(s 1 )F(s 2 ) . . . F(si-i)G(si)w(si) . . . w(sj) dsi . . . ds 2 ds l 
+ .... (2.64) 

We now multiply both sides of this equation by $>(t, a) and use that £(£) = 
$(t, c)£(£) and the formulas for F and G in terms of <&,B,E, to conclude that 
there is a Volterra expansion 

£(*)=£)/ f L ... f ' W l {t,s 1 ,...,s l )uj(s 1 )...uj(s l )ds l ...ds 1 (2.65) 

where W^(t,Si, . . • , Sj) := 

*(t, *i)£(*i)$(«i, 5 2 )-B(s2) • • ■ E(st-i)Q(8i- U si)B(si) . (2.66) 

The case when £(<r) = i° ^ can be handled as follows: Introducing the 
variable 

there results for £ the equation 

^(t) = A(t)ffl + u(t)E(t)S(t) + B(t)u(t) 

having £(o~) = 0, where 

B(t) := E(t)&(t,a)x° + B(t) , 

and a series can be obtained for £ as above. 

Now it is possible to obtain a representation for the output y as 

y(t) = C(t)*(t,v)x° + V(t), 

where V(t) is given by 

J2 f ( * ••• / ' C(t)W l {t,s 1 ,...,si)u;(s 1 )...u;(si)dsi...ds 1 (2.67) 

for the series obtained for £. 

Exercise 2.11.2 The transmission of FM signals is based on the modulation 
of frequency based on an input voltage. A simple circuit generating such mod- 
ulated signals is the one satisfying the differential equation 

y(t)+ [A 2 + u(t)} y(t) = , 
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which corresponds to a harmonic oscillator with frequency controlled about the 
nominal value A. We model this as a time-invariant bilinear system 

* = (_° A2 l)x + „(_° 1 ° Q ) X (2.68) 

y = (1 0)x 

with X = E 2 , It = E, and )) = E. (The "B" matrix is identically zero.) When 
u = one has a standard harmonic oscillator, and 

, , / sin At 
\ A cos At 

is a solution, with 

'0 



*(0) - , A 

Fine? the first three terms of the Volterra representation of y(t) for (2.68), with 
this initial state. □ 



2.12 Notes and Comments 

Basic Definitions 

Various definitions of "system" or "machine" have been proposed in the liter- 
ature, all attempting to model basically the same notion, but differing among 
themselves in mathematical technicalities. Some early references are [114], [230], 
and [444], but of course similar transition formalisms (with no controls nor 
outputs) have long been used in the theory of dynamical systems. Category- 
theoretic definitions also have been suggested, which make the definition of 
"linear" system or certain classes of "topological" systems very easy; see, for 
instance, [22] for pointers to that literature, as well as [314], Section 8.5. The 
area called general systems theory concerns itself with abstract systems defined 
by transition maps, typically with no extra algebraic or topological structure; 
see, for instance, [300] and [429]. 

Recently there has been a resurgence of interest in control theory in the 
use of automata models that had originated in computer science, logic, and 
operations research. In most of these studies, time corresponds not to "real" 
clock time but to instants at which the system changes in some special manner, 
as in queuing models when a new customer arrives. Work in this area falls 
under the general label of "control of discrete-event systems'"; some references 
are [90], [405], and [433]. An approach combining discrete-event ideas together 
with more classical linear design, in the context of a special-purpose computer 
language, has been recently proposed in [46]; this paper models some of the 
logical operations as discrete-time systems over a finite field. More generally, 
the area of hybrid systems has become active in the mid 1990s, see for instance 
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the conference proceedings [15]; one possible approach to combining automata 
and linear systems was studied in [362] (see also the author's paper in [15]). 

A conceptually very different foundation for system and control theory has 
been also proposed. Rosenbrock in [331] emphasized the fact that physical prin- 
ciples sometimes lead to mixtures of differential and algebraic equations, and 
state space descriptions are not necessarily the most appropriate; implicit dif- 
ferential equations are then used. Related to this is the literature on singular 
systems, or descriptor systems, defined (in the linear case) by equations of the 
type Ex = Ax + Bu with the matrix E not necessarily invertible; see, e.g., 
[83], [94], [192], and [277], as well as [82] for the related differential equation 
theory. The recent work of Willems has carried out this reasoning to the limit 
and proposed a completely different formalism for systems, based on not distin- 
guishing a priori between inputs and outputs. Instead of "input/output" data, 
one then has an "observed time-series" which summarizes all i/o data without 
distinguishing between the two; the papers [309], [424], [425], and [426] provide 
an introduction to this literature. 

The definitions given may be generalized to include what are often called 
multidimensional systems, for which the time set T is R fe or Z fe , with k > 1. 
(The terminology is most unfortunate, since we normally use the term "dimen- 
sion" to refer to the number of dependent variables in the equations defining 
the dynamics, rather than to the number of independent variables.) Multidi- 
mensional systems are useful in areas such as picture processing or seismic data 
analysis, where different space directions may be thought of as "time" coordi- 
nates. Multidimensional systems are considerably more difficult to handle, since 
the difference and differential equations that appear with ordinary systems are 
replaced by partial difference and differential equations. For references to this 
area, see the book [57]. 

There is also a variation of the definition of system that allows for the possi- 
bility of the readout map h to depend directly on ui(t). In other words, one has 
h: TxXxU—>y. In the context of such more general systems, one would call 
state-output systems those that we have defined here. The use of non-state 
output systems causes some technical difficulties which are avoided with our 
definition, but one could model such more general systems using pairs (S,a), 
where £ is a system with outputs as defined in this Section and a : U x ^ — > y 
is a map. 

In practice, it may often happen that control of a large system can only be 
achieved through local effects; this gives rise to the theory of large scale systems 
and corresponding decentralized control objectives; see [350] for a detailed treat- 
ment of many of the issues that arise there, including the use of decomposition 
techniques, as well as many references. 

Continuous-time systems x = f(x, u) can be studied as differential inclusions 
([26], [111], [93]). In general, given a set- valued map F : X — > 2 R , that is, a map 
that assigns to each point xinlC R™ a subset F(x) C R n , a solution of the 
differential inclusion x € F{x) means, by definition, an absolutely continuous 
function x : X — > X, defined on some interval, so that x(t) G F(x(t)) for almost 
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all t G 1. To a system (£) x = f(x, u) one may associate the differential inclusion 
with F-£,(x) := {f(x,u) \ u e It}, and paths of the system become solutions of 
the latter. Conversely there is a rich literature dealing with selection theorems 
which, under appropriate regularity assumptions on F^, guarantee that every 
solution of x = -Fe (x) arises in this manner, from some control u. 

I/O Maps 

There are sometimes cases in which the input affects the output with no delay, 
for instance in the "i/o behavior" of more general systems such as those men- 
tioned above [h depends in u) or in dealing with a memoryless i/o map, a map 
induced by an arbitrary mapping 

h:U-y}) 

between any two sets by the obvious rule 

\{T,<j,uj){t) :=h{uj{t)). 

Unfortunately, there are technical difficulties in formulating this consistently in 
terms of response maps. For instance, when treating continuous-time systems 
and behaviors one identifies controls that are equal almost everywhere, and 
this means that the instantaneous value u>(t) is not well-defined. Also, for the 
empty control the output would be undefined, meaning that one cannot really 
talk about "instantaneous measurements" of states. Thus, a concept analogous 
to that of response cannot be used in this case. It is possible, however, to 
generalize the idea of i/o map, where strict causality is now weakened to the 
requirement (causality) that 

A(/i,cr,w|[ CTiM ))(i) = \(t,o-,u))(£) 

only for t G [a, fi) , rather than for all t € [a, fi] . Then a satisfactory theory can 
be developed for causal mappings. We prefer to deal here with the concept of a 
response, or equivalently a (strictly causal) i/o map. 

Linear Discrete-Time Systems 

In control applications, linear systems are of interest mainly in the case in 
which the field is K = M, and sometimes, mostly for technical reasons, K = C 
However, finite fields do appear in communications and signal processing theory, 
as illustrated by the error detection example; in particular the theory of what 
are called convolutional codes is to a great extent the study of linear systems 
over such fields (see, for instance, [118] and [297]). Recently, work has been done 
on systems over fields of formal Laurent series, which provide a good model for 
certain perturbation results; see, for instance, [313]. 

The algebraic theory of linear systems is no more difficult when dealing with 
arbitrary fields than with R or C, and so we will give most results for general K. 
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On the other hand, it is also possible to define linear systems over rings rather 
than fields, with the same definition but using "module" instead of "vector 
space" ; these are useful in various applications such as modeling the effect of 
fixed precision integer arithmetic or delay systems (see remarks below). Some 
results valid over fields generalize, typically with much harder proofs, to systems 
over rings, but many others do not. For references on systems over rings see, 
for instance, [61], as well as the early papers [332] and [333] that started most 
of this research. 

Linear Continuous-Time Systems and Behaviors 

It is also of great interest to consider linear (or nonlinear) infinite dimensional 
systems, also called distributed systems. For instance, transmission delays give 
rise to what are often called delay-differential or hereditary models such as 

x(t) =x(t- l) + u{t), 

which reflects the effect of a retarded feedback; or one may study systems defined 
by partial differential equations, such as a heat equation with boundary control. 
Many such examples can be modeled by equations of the type x = Ax + Bu 
evolving in Banach spaces. There is a rich and extensive literature on these 
matters; see, for instance, [281] for optimal control, as well as the books [107], 
[133], or [147], or the paper [437], for more "system theoretic" questions, each 
using a somewhat different approach. 

Sometimes it is possible to model usefully infinite dimensional systems as 
"systems over rings of operators." For instance, one may write the above delay 
system as the "one dimensional" system 



where "cr" is a shift operator on an appropriate space of functions. Viewing 
this as the linear system (cr, 1) over the polynomial ring M.[a] permits using 
techniques — like Pole Shifting — from the theory of systems over rings, resulting 
in constructive synthesis procedures. This approach is due to Kamen; see, for 
instance, [61], [62], [79], [81], [233], [234], [305], [349], [357], [396], and [397] for 
more on the topic and related results, and [74] for the study of certain systems 
defined by partial differential equations in this manner. 

Volterra Expansions 

The Volterra expansion can be generalized to bilinear systems with more con- 
trols, and locally or in asymptotic senses to more general classes of nonlinear 
systems; see, for instance, [336] for details on this and the above example. (One 
approach for general nonlinear systems relies in first approximating nonlinear 
systems by bilinear ones, a process analogous to taking truncations of a Tay- 
lor expansion. There is also an alternative approach to bilinear approximation, 
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valid in compact-open topologies of controls and based on the Stone- Weierstrass 
Theorem; see [381].) 

Volterra series are important because, among other reasons, they exhibit the 
various homogeneous nonlinear effects separately. In the context of identifica- 
tion, they allow the separation of higher harmonic responses to periodic signals. 
When the original system is linear, every term except that for I = 1 vanishes, 
and the Volterra series (2.65) is nothing more than the variation of parameters 
formula; more generally, the solution of the linearized system along £ = 0, u) = 
is obtained by simply dropping all terms except the first one. It is also possible 
to give Volterra expansions for smooth discrete-time systems, as also discussed 
in [336]. 

Volterra series were one of the most popular engineering methods for non- 
linear systems modeling during the 1950s. Their theoretical study, especially 
in relation to state space models, was initiated during the early 1970s; see, for 
instance, [69] for a survey of the early literature. 

An alternative approach to Volterra series is based on what are called gen- 
erating series, or Fliess series. These are formal power series that essentially 
correspond to expanding the "kernels" Wi, and can be interpreted in terms of 
mappings from jets of germs of differentiable inputs into jets of outputs. They 
are closely related to work on the representation of solutions of differential equa- 
tions, as in [89]; see [140] and references therein, as well as [199], [373], [241], 
and the excellent book [327]. 



Chapter 3 

Reachability and 
Controllability 



3.1 Basic Reachability Notions 

In all of the definitions to follow, £ = (T, X, 11, <ft) is an arbitrary system. 
Definition 3.1.1 An event is a pair (x,t) G X x T. 

• The event (z,t) can be reached from the event (x,<r) iff there is a path 
of £ on [a, r] whose initial state is x and final state is z, that is, if there 
exists an u) € VS '^ such that 

4>(t, o~,x,u>) = z . 

One says also that (x, it) can be controlled to (z, r). 

• // x, z € X, T > € T, and there exist cr, r € T wii/i t — a = T such 
that (z,t) can be reached from (x,a), then z can be reached from x in 
time T . Equivalently, x can be controlled to z in time T. 

• z can be reached from x (or x can be controlled to z) if this happens 
for at least one T . □ 

Note that, by the identity axiom in the definition of system, every event and 
every state can be reached from itself. Also, note that if £ is time- in variant, 
then z can be reached from x in time T iff (z, t + T) can be reached from (x, t) 
for any t 6 T; thus, there is no need to consider explicitly the notion of event 
for time-invariant systems. 

We use the notation (x, it) "-a (z, t) to indicate that the event (z, r) can be 
reached from (x,cr). Note that the inequality a < r is implicit in this notation. 
For states, we write x^f z if z can be reached from x in time T, and just x ~> z 
to indicate that z can be reached from x. 

81 
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Lemma/Exercise 3.1.2 Prove the following statements: 

(a) If {x,a)^> (z,t) and (z,t) ~> (j/,/x), then (x,o-)~> (y,/i). 

(b) If (x,cr) ~> (y, n) and if er < r < /i, then there exists a 2 € X such that 
(x, ct) -^ (z, t) and (z, r) ~> (y, /i). 

(c) If x "^f y for some T > and if < t < T, then there is some z G X such 
that x^z and z y* y. 

t T-t y 

(d) If x^f z, z~^>y, and E is time-invariant, then x rv* y. 

(e) If x ~» 2, z ^^ y, and E is time-invariant, then x ~> y. □ 
Exercise 3.1.3 Give examples to show that: 

• Properties (d) and (e) in Lemma 3.1.2 may be false if E is not time-invar- 
iant, and 

• Even for time-invariant systems, it is not necessarily true that x ~> z 
implies that z ~> x (so, "~>" is not an equivalence relation). □ 

Remark 3.1.4 Let E be a continuous-time system as in Definition 2.6.7 and 
let <t < t. With the present terminology, Lemma 2.6.8 says that (x, er) r ^> 
(z,t) for the system E iff (z,a) ~> (x,r) for the system E~ +T . This remark 
is sometimes useful in reducing many questions of control to a given state to 
analogous questions (for the time-reversed system) of control from that same 
state, and vice versa. □ 

Recall that a linear system is one that is either as in Definition 2.4.1 or as 
in Definition 2.7.2. 

Lemma 3.1.5 If E is linear, then: 

(a) If (xi,a) ~> {zi,t) and (x2,c) ~> (22, r), then, for each r G K, (xi + 
rx 2 ,cr) ~~> (zi + r^ 2 ,r). 

(b) (x, er) ~> (z,t) iff (0, a) ~> {z — 4>{t, a, x, 0),t). (Here denotes the control 
w = 0.) 

(c) If E is time-invariant, x^f z iff O'if (z — (f>(T, 0, x, 0)). 

(d) If E is time-invariant, Xi "^f z\ and X2 'if zi imply that 
(xi + rx 2 ) ^ (zi + rz 2 ) for all r£l. 

Proof. Part (a) follows by linearity of <f>(r, a, •, •) (see Lemmas 2.4.2 and 2.7 A), 
and part (d) is a consequence of this. The equivalence 

4>(t, ct,x,uj) = z iff z — 4>{t, a, x, 0) = <J){t, a, 0, lu) 

implies both parts (b) and (c). I 
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Definition 3.1.6 The system £ is (completely) controllable on the inter- 
val [a, t] if for each x, z G X it holds that (x,a) ~> {z,t). It is (completely) 
controllable in time T if for each x,z G X it holds that x^ z. It is just 
(completely) controllable if x ~^ z for all x, z. □ 

Lemma 3.1.7 Let £ be a linear system, and pick any a, r, T G T. 

(a) £ is controllable on [cr, t] iff (0, cr) ~> (y, r) for all y G X. 

(b) If £ is time- in variant, then £ is controllable in time T iff O^f y for all 
y G X. 

(c) If £ is continuous-time, then it is controllable on [a, r] iff (x, cr) ~> (0,r) 
for all i£l. 

(d) If £ is time-invariant and continuous-time, then £ is controllable in time 
Tiff x ^0 for allccG X. 

(e) The conclusions in (c) and (d) hold also if "continuous-time" is replaced 
by "discrete-time and A{k) is invertible for all k G [cr, r]" (in (d), U A is 
invertible" ) . 

Proof. To prove part (a), pick any events (x,a) and (z, r). Let 

y:= z- 4>{t,o-,x,0) . 

By assumption, (0,c) ~> (y,r). Thus, by Lemma 3.1.5(b), (x,a) ~~* {z,t), as 
desired. Assume now that £ is time-invariant and that 

for all y € X. Then, (0, 0) ~> (j/, T) for all y G X, so by part (a) £ is controllable 
(on [0,T]), and this establishes part (b). 

Assume now that £ is continuous-time, or that it is discrete-time and the 
invertibility condition in (e) holds. Pick any t/gl. By part (a), we must prove 
for (c) that (0, cr) ~> (y, r). We claim that there is some x G X such that 

<l>{T,a,x,0) = -y. 

In continuous-time we may take x := — <l>(cr, r)y, and in discrete-time we take 
x := —P~ l y, where P is the linear transformation 

$(r, s) = A{t - 1)A{t -2)...A{a+ l)A{<j) , 

which is invertible under the assumption that each A(k) is. By hypothesis, 
(x,o~) ~> (0,t). So by Lemma 3.1.5(b), 

(0, a) -^ (-0(t, cr, x, 0), t) = (y, r) , 
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as desired. Finally, part (d) follows from (b) because the hypothesis gives now 
that (x, 0) ~> (0, T) for all x € X. I 

The property in (a) and (b), that any state be reachable from zero, is some- 
times called reachability. We proved that, for linear systems, reachability and 
controllability are equivalent. Note also that the property in (c), (d), and (e), 
that one be able to control every state to the origin, is called "controllability" by 
some authors (a more precise term, also used sometimes, is null- controllability) . 
We prefer to reserve the term controllability for the property defined in 3.1.6. 

Remark 3.1.8 Part (e) of the above result illustrates a principle that is rather 
general in control theory: Finite dimensional smooth discrete-time systems with 
"reversible" or "invertible" dynamics tend to have a theory very analogous to 
that of continuous-time systems. (Essentially, because group actions can be 
associated to systems for which the maps </>(r, o~,-,ui) are invertible.) Except for 
the algebraic structure theory of time-invariant linear systems, general (non- 
invertible) discrete-time systems behave in a much more complicated manner 
than continuous-time systems. One way in which distributed (i.e., infinite di- 
mensional) continuous-time systems behave very differently from continuous- 
time (finite dimensional) systems is precisely in the sense that they tend not 
to have invertible dynamics; for instance, a system that evolves according to a 
diffusion equation will smooth out initial conditions. □ 

3.2 Time-Invariant Systems 

Throughout this section, unless otherwise stated, S is an arbitrary time-invar- 
iant system. 

Definition 3.2.1 Let T G T and igl The reachable set from x in time 

T is 

TZ T {x) :={zeX\x^z}. 

The reachable set from x is 

K{x):= (J K T {x) = {zeX\x^ z}. 
TeT + 

If S is a subset ofX, we also write 

K T (S) := |J K T (x), 

n(S) - |J n(x) , 

x£S 

for the sets reachable from the subset S. □ 
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Note that 1Z°(S) = S for all subsets S, and that E is controllable (respec- 
tively, controllable in time T) iff 1Z(x) (respectively, 1Z T (x)) = X for all iGl 
From Lemmas 2.4.2 and 2.7.4, we have the following: 

Lemma 3.2.2 If E is linear, then 1Z T (0) is a subspace, for all T g T. □ 

If E is linear and a; is now an arbitrary state, the equality 

4>{T, 0, x, u) = 4>(T, 0, x, 0) + <f>(T, 0, 0, ui) 

shows that 

TZ T {x) = (f>(T, 0, x, 0) + TZ T {0) 

and hence that lZ T (x) is a linear submanifold (translation of a subspace) in X. 
Lemma 3.2.3 For each t, s € T, 

K s+t {x)=K s {K t {x)) 

for each iGl 

Proof. Lemma 3.1.2, Part (b) implies that lZ s+t (x) C 7?. s (7?.*(a;)), while the 
reverse inclusion follows by Part (d). I 

It is of interest to know whether reachability can be tested in finitely many 
steps. In automata theory, one frequently uses the following observation: 

Lemma 3.2.4 Assume that E is a discrete-time system and card (X) = n < oo. 
Then, 

|J TZ k {x) = K(x) 

{fe=0,...,n-l} 

for all iGl 

Proof. Pick x and a z € TZ(x). Assume that k is the smallest integer such that 
x~"? z, and assume by way of contradiction that k > n. By part (c) of Lemma 
3.1.2, there exist elements 

x = z , Z\ , . . . , Zfc = z 

with 

z { y z i+ i,i = 0,...,fc- 1 . 

Since k > n, there are < i < j < k such that z% = Zj. Since 

x^ z* and Zi?~*.Z. 
i ' J k-j 

it follows by part (d) of Lemma 3.1.2 that x^f z,l = i + k — j < k, which 
contradicts minimality of A;. I 
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Recall the notion of equilibrium state x £ X; for a time-invariant system, 
this is equivalent to the requirement that there exists a u £ IX such that for 
all T £ 7+, 4>(T, 0, x, u>) = x, where lu = u. For discrete-time systems this is 
equivalent to 

P{x, u) = x 

and for continuous-time systems to 

f(x,u)=0. 

If x is an equilibrium state, then x £ 1Z T (x) for all T. Thus, for all S £ 7+, 

K s {x) CK s {K T (x)), 

and so by Lemma 3.2.3: 

Corollary 3.2.5 If x is an equilibrium state, then 

K s {x) C K s+T {x) 

for each S,T £ 7+ . □ 

Lemma 3.2.6 Let a; be an equilibrium state. If there are any S,T £ T + ,T > 0, 
such that 

n s (x) = n s+T (x) , 

then necessarily 1Z S (x) = IZ(x). 
Proof. First note that 

n s+kT {x) = n s {x) 

for all positive integers k. This can be proved inductively: The case k = 1 is 
given, and 

n S+(k+l)T {x) = n T {n S+kT {x)) 

= TZ T {R s {x)) 

= n s+T (x) 

= K s {x), 

where the second and last equalities hold by inductive hypothesis. Now assume 
given any z £ TZ(x). So there is some t £ T so that z £ iV'ix). Find an integer 
k such that S + kT > t. By Corollary 3.2.5, z £ K s+kT {x) = K s {x). I 

Corollary 3.2.7 Assume that x is an equilibrium state, that X is a vector 
space over a field IK with dimX = n < oo, and that lZ T (x) is a subspace for 
each T £ T + . (By Lemma 3.2.2, this always happens if S is a finite dimensional 
linear system and x = 0.) Then: 
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(a) IfT = Z, K n {x) =K(x). 

(b) If T = R, then TZ e {x) = K(x) for all e > 0. 
Proof. Pick any strictly increasing sequence 

= t < n < t 2 < . . . < T n+1 

of elements of T, and let 

X, :=7e Ti (:r) 

for each i. If it were true that Xj were properly included in Xj+i for all these i, 
then it would follow that dim Xj+i > dim Xj for all i, and hence that dim X n +i > 
n + 1 > dim X, a contradiction. Thus, in each such chain of reachability sets 
there is some i such that Xj = X i+1 . By Lemma 3.2.6, we conclude that 

K Ti (x) =K(x). 

Now in case (a) we apply the above argument with t% = i, and in case (b) we 
use instead t% = ie/n. I 

The conclusion that, for linear continuous-time systems, if one can reach z 
from 0, then one can do so in arbitrarily small time, is non-intuitive. Of course, 
this can never happen in a physical system. Underlying the above proof is the 
fact that the control- values space is all of K m . In order to obtain smaller e, 
one will in general need larger values for the controls (see, for instance, Exercise 
3.5.7). Later, in the Chapter on optimal control, we will state an optimization 
problem that imposes a cost associated to the magnitudes of controls. 

Lemma/Exercise 3.2.8 Let 

C T (x) = {zeX\zyx}, C(x)= |J C T {x). 

T£T+ 

Prove Lemmas 3.2.3, 3.2.4, and 3.2.6, and Corollaries 3.2.5 and 3.2.7 for the 
statements that result when "C" is substituted for "72." throughout. □ 

Exercise 3.2.9 Consider a discrete-time bilinear system x(t + 1) = u(t)Ex(t), 
with X = R 2 ,ll = ]R, and 

1 0~ 



E 



Show that 

T fl\ I (xi 



K 1 



x 2 



Tx\ = x 2 



for each positive integer T. This is a subspace for each such T, but R T ( ) 7^ ^(n) 
for all T. Does this contradict Corollary 3.2.7? □ 
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Corollary 3.2.10 (a) An n-dimensional discrete-time linear system is con- 
trollable if and only if 

iz(o) = iz n (o) = x. 

(b) A continuous-time linear system is controllable if and only if 

11(0) = 1Z £ (0) = X for all e > 

if and only if C(0) = C 6 (0) = X for all e > 0. 

Proof. If E is controllable, then IZ(x) = X for all x G X, so by Corollary 
3.2.7 it holds in particular that !Z n (0) = X in the discrete-time case, and that 
7^ £ (0) = X for all e > in the continuous-time case. Conversely, if 1Z T (0) = X 
for some T > 0, then E is controllable, by Lemma 3.1.7(b). In continuous-time, 
C T (0) = X implies the same conclusion, by part (d) of the same Lemma. I 

Exercise 3.2.11 Give an example of a discrete-time (time-invariant) linear sys- 
tem of dimension 1 for which C(0) = X but 1Z(0) = {0}. □ 

Let E be an n-dimensional discrete-time time- invariant linear system (A, B). 
From the formula 

n 

4>(n, 0, 0, w) = Y^ A*- l Buj(n - i) , 
i=i 

it follows that z is in !Z n (0) iff it is in the image of the linear map 



R = R(A, B) := [B, AB, A 2 B, ..., A n - l B] (3.1 



which maps 

U n ->X 

sending 

n 

(ui,...,u n ) i-> ^ A t ^ 1 Bu l . 

i=l 

When IX = K m and X = K™, we identify R(^4, B) with an n by nm matrix whose 
columns are the columns of B, AB, . . . , A n ~ 1 B in that order. By Corollary 
3.2.10, E is controllable iff the image of the linear map R is all of X, and this 
happens iff its rank (i.e., the dimension of its image) is n. Thus: 

Theorem 2 The n-dimensional discrete-time linear system E is controllable if 
and only if rank R(A, B) = n. □ 

Note that from the Cayley-Hamilton Theorem one knows that 

A n = c^I+azA+.-. + anA"- 1 (3.2) 
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where 

X a (s) = det(sl - A) = s n - anS 11 " 1 - . . . - a 2 s - c*i (3.3) 

is the characteristic polynomial of A. Multiplying both sides of (3.2) by A, and 
using again (3.2) to substitute for the A n that results in the right-hand side, it 
follows that A n+l is a linear combination of the matrices A 1 ,i = 0, . . . , n — 1. 
Recursively, the same holds for all powers of A. Thus, for any vector v, the span 
of {A l v,i > 0} coincides with the span of {A l v,i < n}. Applying this to each 
of the columns of B, we conclude that the condition rankR(A, B) = n fails if 
and only if there is some n-dimensional row vector p ^ such that 

P A l B = for all i > , (3.4) 

which means that there exists a linear function (namely, x \— » px) on the state 
space that vanishes at all states reachable from the origin. 

If now E is an n-dimensional continuous-time system, we still may consider 
the matrix R(A, B) as in (3.1). A fact that is at first sight remarkable - 
and which is ultimately due to the common properties shared by difference 
and differential time-invariant linear equations — is that the same algebraic 
condition as for discrete-time is necessary and sufficient for continuous-time. 
We now prove this fact. (Later, cf. Remark 3.5.22, we derive it from a general 
theorem about time-varying systems.) The critical observation is that, for any 
row vector p, 

pe tA B = y p A i B- = 
*—* v. 

if and only if (3.4) holds, because of analyticity of the exponential. 

Theorem 3 The n-dimensional continuous-time linear system E is controllable 
if and only z/rankR(A, B) = n. 

Proof. If the rank condition fails, there exists some row vector p ^ so that 
(3.4) holds. Since every element of 7^.(0) = TZ (0) has the form 

x = f e {l - t)A Bu{t)dt = f e tA Biv(l-t)dt, (3.5) 

./o Jo 

also px = for all such x. Thus, 7^.(0) ^ X, and the system is not controllable. 
Conversely, if controllability fails, then there is some p ^ so that px = 
for every x in the subspace 7^.(0). In particular, consider the control 

w(t) := B*eS 1 ~ t '> A 'p* 

on the interval [0, 1], where "*" indicates conjugate transpose. From (3.5), 



l 

tA* * 



= px = / pe tA BB*e tA p* dt 



B*e l 



dt. 
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and hence, pe' B = 0. I 

Take as an illustration the linearized pendulum (harmonic oscillator). With 
the matrices in equation (2.31), 

and the system is controllable. This means that every configuration of position 
and velocity can be changed into any other such configuration through the 
application of a suitable control. We later compute an explicit example of such 
a control. 

For the original nonlinear pendulum, the use of feedback transformations as 
discussed in Chapter 1 allows establishing that 

xi = X2, ±2 = — sinxi + u 

is also controllable; it is only necessary to modify whatever control is used to 
transfer x to z in the linear system ii = x 2 , i 2 = u by addition of the term 
sin£i(£) evaluated along the ensuing trajectory. This is just a property of this 
simple example; in general, for nonlinear systems controllability is far more 
difficult to characterize. The linearization principle for controllability studied 
later is useful in reducing questions of "local" controllability to the linear case, 
however. 

Exercise 3.2.12 Assume that the body of an airplane is slanted <f) radians (its 
"pitch angle") with respect to the horizontal. It is flying at a constant (nonzero) 
ground speed of c meters per second, and its flight path forms an angle of a 
radians with the horizontal (for a > the plane is gaining altitude, and for 
a < it is descending). Denoting the plane's altitude in meters by h and 
assuming the angles are small, the above quantities are related by the following 
linearized differential equations: 

a = a(<f> — a) 

if> = —w(<f> — a — bu) 

h = ca 

where to > is a constant representing a natural oscillation frequency and a, b 
are positive constants. (See Figure 3.1(a).) The control u is proportional to the 
position of the elevators. (Elevators are movable surfaces located in the tail. A 
more complete space and orientation model would include other such surfaces: 
For instance, the rudder, also in the tail, provides directional control or "yaw," 
and the ailerons in the wing create opposing torques to affect lateral attitude 
or "roll." In addition, engine thrust affects speed.) 

Model the above as a linear system E with n = 4, m = 1, using Xi = a, 
x 2 = 4>, x 3 = <f>, and X4 = h. Prove that E is controllable. □ 
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Figure 3.1: (a) Airplane example, (b) Inverted pendulum. 



Exercise 3.2.13 Consider a system consisting of a cart to the top of which an 
inverted pendulum has been attached through a frictionless pivot. The cart is 
driven by a motor which at time t exerts a force u(t), taken as the control. (See 
Figure 3.1(b).) We assume that all motion occurs in a plane, that is, the cart 
moves along a straight line. We use </> to denote the angle that the pendulum 
forms with the vertical, 5 for the displacement of the center of gravity of the cart 
with respect to some fixed point, F > for the coefficient of friction associated 
with the motion of the cart, g for the acceleration of gravity, I > for the length 
of the pendulum, M > for the mass of the cart, and m > for the mass of 
the pendulum, which we'll assume is concentrated at the tip. (If the mass is 
not so concentrated, elementary physics calculations show that one may replace 
the model by another one in which this does happen, using a possibly different 
length I. We allow the case m = to model the situation where this mass is 
negligible.) 

Newton's second law of motion applied to linear and angular displacements 
gives the two second order nonlinear equations 



(M + m)6 + ml<f> cos </> — mlcj) 2 sin <f> + FS 



F 


mg 


M 


M 








F 


g(m+M) 


IM 


IM 




92 3. Reachability and Controllability 

and 

l(j) — g sin cj) + 5 cos <f> = . 

We shall only be concerned with a small angle <ft, so we linearize the model 
about <j> = 0. This results, after taking x\ = S, Xi = S,x^ = (f>,X4 = <f>, in a 
linear system £ with n = 4, m = 1 and matrices as follows: 



B 



Prove that £ is controllable. (Controllability holds for all possible values of the 
constants; however, for simplicity you could take all these to be equal to one.) 
This example, commonly referred to as the "broom balancing" example, is 
a simplification of a model used for rocket stabilization (in that case the control 
u corresponds to the action of lateral jets). □ 

3.3 Controllable Pairs of Matrices 

Note that the controllability conditions are exactly the same, in terms of the 
matrices (A,B), for the discrete and continuous case; so we are justified in 
making the following definition for pairs of matrices: 

Definition 3.3.1 Let K be a field and let A e K nxn ,B e K nxm , for positive 
integers n, m. The pair (A, B) is controllable or reachable if rank R(A, B) = 
n. □ 

We call R = R(A, B) the reachability (or controllability) matrix of (A,B); 
the image of the corresponding map, i.e., the column space of R, is the reachable 
(or controllable) space of (A,B), and we denote it by 

n(A,B) 

or just TZ. For discrete-time or continuous-time linear systems, TZ is then the 
same as the set of states reachable from the origin, as is clear for discrete-time 
and follows from the proof of Theorem 3 (p. 89) for continuous-time. Whenever 
we refer below to a "pair" (A, B), we mean an (A, B) as above. The Cayley- 
Hamilton Theorem implies the following observation: 

Lemma 3.3.2 Let (A,B) be a pair as in Definition 3.3.1, and let 

bj := jth column of B, j = 1, . . . ,m. 

Then the controllable space TZ(A, B) is the span of 

{A i b j ,i>0,j = l,...,m}. 

Thus, 1Z is the smallest yl-invariant subspace of X = K™ that contains the 
columns of B. In particular, (A, B) is controllable iff this span is all of X. □ 
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When m = 1 this says that controllability of (A, B) is equivalent to the 
requirement that B, seen as a (column) vector in K™, be cyclic for the matrix 
A. In this sense controllability generalizes the classical notion of cyclicity in 
linear algebra. 

We let GL(n) denote the group of all invertible n x n matrices over K. 
The following result, often called the Kalman controllability decomposition, is 
extremely useful in giving simple proofs of facts about controllability. 

Lemma 3.3.3 Assume that (A, B) is not controllable. Let dim 1Z{A, B) = r < 
n. Then there exists a T € GL{n) such that the matrices A := T~ l AT and 
B := T~ l B have the block structure 



A, A 2 \ fi_(B 1 

o a 3 b -{o 



(3.6) 



where A\ is r x r and B\ is r x m. (If r = 0, the decomposition is trivial, and 
Ai, A2, B\ are not there.) 

Proof. Pick any subspace S such that 

K © S = K n 

and let {v\, . . . , v r } be a basis of 1Z and {u>i, . . . , w„_ r } be a basis of S. Then, 
with 

T := (Vl,.. .,V r ,Wi,... ,W n - r ) , 

the desired forms for A and B follow from the facts that TZ is A-invariant and 
that it contains the image of B. I 

Lemma/Exercise 3.3.4 Prove that (A\,Bij is itself a controllable pair. □ 

For a continuous-time system (A,B), we can interpret the above decompo- 
sition as follows: With the change of variables 

x(t) := Tz(t) 

the equations x = Ax + Bu are transformed into 

k\ = A\Z\ + A 2 z 2 + Biu 
Z2 = A 3 z 2 

where Z\,Z 2 are r- and (n — r)-dimensional respectively. It is then clear that 
the z 2 component of the state cannot be controlled in any way. An analogous 
observation applies to discrete-time systems. Note that the "only if" part of 
Theorem 3 (p. 89) is an immediate consequence of this decomposition result. 
The characteristic polynomial of A splits as 
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where 



X. 



y . and y 



X , 



Note that A\ is a matrix representation for the restriction of A to 1Z, so \ 
is independent of the choice of basis on TZ. The same is true therefore for 
X = Xa/x • (Independence of choice of basis for the latter also follows from 
the fact that A3 is a matrix for the linear map induced by A on the quotient 
K. n /1Z and hence is uniquely defined up to similarity.) If r = 0, we make the 
convention that y =1. 

Definition 3.3.5 The polynomials \ an d X ar e, respectively , the control- 
lable and the uncontrollable parts of the characteristic polynomial \ A (with 
respect to the pair (A, B)). □ 

Sometimes one refers to the eigenvalues (or the eigenvectors) of these as the 
"controllable modes" and "uncontrollable modes," respectively, of the system 
(or matrix pair) (A,B). 



Exercise 3.3.6 Prove that if 
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1 and A is a diagonal matrix 
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then (A, B) is controllable iff A^ 7^ Xj for each i 7^ j and all b l 7^ 0. □ 

We will refer to the following result as the Hautus Lemma: 

Lemma 3.3.7 Let IK be the algebraic closure of the field K. (In most applica- 
tions, K = 1 or C, so K = C.) The following properties are equivalent for the 
pair (A,B): 

(a) (A, B) is controllable. 

(b) rank [XI -A,B] = n for all A € K. 

(c) rank [XI — A, B] = n for each eigenvalue A of A. 

Proof. Note first that (b) and (c) are equivalent, since the first n x n block of 
the n x (n + m) matrix [XI — A, B] already has full rank whenever A is not an 
eigenvalue of A. Also, since controllability is characterized by a rank condition, 
and hence by the nonvanishing of certain determinants, the pair (A, B) is con- 
trollable iff it is controllable as a pair over the field K. It follows that we may 
assume that K = K. 
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We prove next that (a) implies (b). Assume that the rank is less than n for 
some A G K. Thus, the row space of [XI — A, B] has dimension less than n, so 
there exists some nonzero vector p G IK™ and some A such that 

p'[XI-A,B] =0. (3.7) 

Thus, p'A = Xp' (A is a left eigenvalue of A), and p'B = 0. Therefore also 
p'A k B = X k p'B = for all k, and hence 

p'R(A, B) = 0, 

contradicting controllability. 

Finally, we establish that (b) implies (a). Assume that (a) does not hold, 
so there is a decomposition as in (3.6), with r < n. Let X,v be an eigen- 
value/eigenvector pair of the transposed matrix A f 3 , so that 

v'(XI-A 3 ) = 0. 

It follows that the nonzero n- vector 

'0 N 



w : 

\ v 

is a left eigenvector of A, with w' B = 0. Hence p := {T')~ l w ^ satisfies 

p'[(XI-A)T,B}=0. 

Since the column space of the matrix [(XI — A)T, B] coincides with that of 
[XI — A, B], also (3.7) holds, and hence (b) cannot be true. I 

Corollary 3.3.8 If (A, B) is controllable, then rank [A, B] = n. D 

Recall that the geometric multiplicity of an eigenvalue A of a matrix A is the 
dimension of the nullspace ker (XI — A). 

Lemma/Exercise 3.3.9 Assume that (A, B) is controllable and that rank B = 
q. Then, the geometric multiplicity of each eigenvalue of A is at most q. □ 

The uncontrollable modes can be characterized elegantly using the Hautus 
condition, as follows. 

Lemma/Exercise 3.3.10 The zeros of \ are precisely the complex numbers 
A for which 

rank [XI - A, B] < n . D 
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It would seem to be natural also to study a slightly more general class of "lin- 
ear" (or more precisely, "affine") systems, namely, the class of non- homogeneous 
linear systems 

x = Ax + Bu + c, (3.8) 

c / (or their analogue in discrete-time). At least for controllable systems 
however, the following simple consequence of Lemma 3.3.7 shows that, except 
for a translation of coordinates in state and input-value spaces, x — > x — xq, 
u — > u — wo, homogeneous linear systems result again. By colVF we denote the 
column space of the matrix W , i.e., the span of the columns of W . 

Exercise 3.3.11 Assume that (A,B) is a pair as before and c £ K" is an 
arbitrary vector. Prove that: 

(a) If there is any A £ K such that c ^ col [XI — A, B], then there is a T £ 
GL(n) such that, with the change of variables z := Tx, equation (3.8) has 
the partitioned form 

Z\ = A\Z\ + Bm, 

z 2 = \z 2 + 1 

(zi of size n — 1 and z 2 scalar) and similarly in discrete-time. It follows 
that the system (3.8) and its discrete-time analogue are not controllable. 

(b) If c £ col [A, B], then there are xq, uq such that 

Ax + Bu + c = A(x — xq) + B(u — uq) 

and if c £ col [I — A, B] there are xq, uq such that 

Ax + Bu + c = A(x — xo) + B(u — uq) + xq . 

Interpret in terms of a differential (respectively, difference) equation for 
x — xq with control u — uq. 

(c) Conclude that a nonhomogeneous linear system as above is controllable if 
and only if the pair (A, B) is controllable and that in that case the system 
is, up to a change of variables, a linear system. □ 

Let K = 1 or C. One important characteristic of the notion of controllability 
is that it is a generic property, in the sense that the set of controllable pairs is 
an open and dense subset of the set of all pairs of a given size. We formulate 
this statement as follows. For each fixed pair of positive integers n, m, let 

S n . m := {(A, B)\A£ K" x ", B £ K" xm } , 

identified with K™ + nm by listing all coefficients of A, B in some fixed order. 
We let S„ m be the subset of S n>m consisting of all controllable pairs. 
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Proposition 3.3.12 If K = R or C, the set S„ m is open and dense in S n>m . 

Proof. The complement S™ m of S^ m is an algebraic subset of S rhm , that is, the 
set of common zeros of a set of polynomials {Pi, . . . , P r } in n 2 + nm variables. 
Indeed, we may take as the Pi the set of all possible determinants of n x n 
submatrices of R(X, Y), where X,Y are matrices of indeterminates. Then 

rankR(A, B) < n, 

that is, 

(A,B)eSl m 

iff all these minors vanish. Note that S% m is closed, since it is the intersection of 
the zero sets of the Pj, each of which is closed by continuity of the Pj. Moreover, 
S% m is proper, since for any fixed n, in there exists at least one controllable 
system. 

It is only left to prove that any proper algebraic subset of a Euclidean space 
K 9 must have a dense complement, or equivalently that this set cannot have any 
interior. Since an algebraic set is an intersection of sets of zeros of polynomials, 
it is only necessary to see that a set of the type {y\P(y) = 0} (P polynomial) 
cannot have a nonempty interior unless P = 0. This is clear by the principle 
of analytic continuation, or explicitly for polynomials simply by noting that all 
derivatives of P must be zero at an interior point, and hence that all coefficients 
of P are zero. (Alternatively, one could argue purely algebraically, using in- 
ductively the fact that a single- variable polynomial cannot have infinitely many 
zeros.) I 

One may also prove that S™ m has Lebesgue measure zero. Genericity of 
controllability means that, if all the coefficients of A and B are experimentally 
measured quantities, then the resulting system in all likelihood will be control- 
lable. However, if a system is near <S" m , it will be "hard" to control, and in 
fact the "true" system we are trying to model may be uncontrollable, with mea- 
surement errors accounting for the discrepancy. It is important in practice to 
estimate how close a system is to being uncontrollable. Various recent papers 
deal with this issue, and numerically robust methods based on this research 
have been designed. In particular, one has the following characterization. 

For any controllable pair (A, B) over C, we let 

6(A,B) :=dist ((A, B),Sl m ) , 
where the distance is measured with respect to the operator norm, i.e., 

dist(P,G) = \\F-G\\ 

for any two matrices P, G of size n x (n + m). Since for each fixed n, m the set 
of uncontrollable systems is closed — in fact it is an algebraic set, as remarked 
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in the above proof — it follows that the distance is always achieved, i.e., there 
is some (A, B) G S™ m so that 

6(A,B) = dist( (A, B), (A, B)) . (3.9) 

This distance is not easy to compute, as it involves a minimization over all 
uncontrollable pairs. On the other hand, we may consider the quantity 

S'(A,B) :=mincr min (AI-A£), 

where <r m i n denotes smallest singular value, which involves just a scalar mini- 
mization. (Singular values are discussed in Appendix A. 2.) We are justified in 
writing "minimum" as opposed to just "infimum" because the function 

f(X):=a min (XI-A,B) 

is nonnegative and continuous (continuity of singular values on matrix entries, 
Corollary A. 4. 5 in Appendix A. 4, using the fact that the rank of (XI — A, B) is 
always n, by controllability), and limm^oo f(X) = oo because 

<x min (AI -A,B) = |A| cr min (i - jA, jB 

and a min (I - \A, \B) -> cr min (7, 0) = 1. 

Proposition 3.3.13 For each controllable pair (A,B), S(A,B) = S'(A,B). 

Proof. Let (A, B) G S™ m be as in (3.9), and write 

A A := A- A, A B := B-B . 

By the Hautus condition, there is some A so that 

[XI -A,B] = [XI -A,B] + [-A A , A s ] 

has rank less than n. By Corollary A. 2. 5 applied to (XI — A, B) and A = 
(-Aa,A b ), 

S'(A,B) < a min (XI-A,B) < \\(-A A ,A B )\\ = \\(A A , A B )\\ = 6(A,B) . 

To prove the other inequality, let A be such that 5'(A,B) = (T m i n (AI — A,B). 
Again by Corollary A. 2. 5, there must exist matrices (A^, A B ) of norm equal 
to fJ m i n (A/ — A, B) so that, defining A, B as above, this is a pair at distance 
a m { n (XI — A, B) from (A, B), which is not controllable. So 

S(A, B) < a min (XI -A,B) = S'(A, B) 

holds too. I 
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Exercise 3.3.14 Let (A, B) correspond to a time-invariant discrete-time lin- 
ear system E. Recall that null-controllability means that every state can be 
controlled to zero. Prove that the following conditions are all equivalent: 

1. E is null-controllable. 

2. The image of A n is contained in the image of R(.A, B). 

3. In the decomposition in Lemma 3.3.3, A3 is nilpotent. 

4. rank [XI — A, B] = n for all nonzero A G K. 

5. rank [I - XA, B] = n for all Ae£ □ 

3.4 Controllability Under Sampling 

When using digital control, inputs are restricted in various ways. As discussed 
at length in Chapter 1 , one way of modeling the restriction to piecewise constant 
controls is via the notion of sampling. It is therefore of interest to know when 
a continuous-time system that is known to be controllable remains so if we 
can only apply controls that are constant on sampling intervals [k6, (k + 1)6]. 
Mathematically, the question is whether the sampled system Em is controllable, 
and the answer will depend on the sampling time 6 as well as the system E. 
The material in Appendix A. 3 is used here. 

If E is a continuous-time linear (still time-invariant) system, u € IX is a 
control value, and 6 > is a real number, the variation of parameters formula 
gives that, for u> = u, 

r s 
4>{6,0,x,uj) = e 5A x + / e {S - s)A Bu ds 

Jo 
= Fx + Gu, 

where 

r s 

F := e 5A , G := #B , A^ := / e {5 - s)A ds. (3.10) 

Jo 

So Em is the discrete-time time- in variant linear system (F^), and it is con- 
trollable iff 

R(F,G) = [G,FG,...,F n - 1 G] 

= A^[B,e 5A B,...,e^ n -^ 5A B] 

= A {s) R(e SA ,B) (3.11) 

has rank n. The second equality follows from the fact that the matrices A^ > 
and e commute, being both functions of A. Note that A^ ' = f(A), where / 
is the entire function 

<5 °° xn+1 n S 



f(s) = / e«-*'dt = £ 



5 n+1 „ e Ss - 1 



(n+ 1)! 

n=0 V ' 



S 
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Since sf(s) = e — 1, one also has the useful identity 

AA^ = e 5A -I. 

We shall say that the continuous-time system E is <5-sampled controllable 
if the discrete-time system Em is controllable. Of course, ((-sampled control- 
lability for any 5 > implies controllability, since for any states x, Z, x ~~> z in 
Em also implies that x ~> z in the original system E (using piecewise constant 
controls). The interest is in determining conditions for the converse to hold. 
From equation (3.11) it follows that E is <5-sampled controllable iff both AS> 
and R(e , B) have rank n. 

Lemma 3.4.1 E is (5-sampled controllable if and only if the pair (e , B) is 
controllable and A has no eigenvalues of the form 2kiri/5 for any nonzero integer 
k. 

Proof. By the above remarks, it suffices to prove that A^ ' is invertible if and 
only if the eigenvalue condition holds. This follows from the Spectral Mapping 
Theorem, since its eigenvalues are the possible values /(A), where / is the above 
function and A is an eigenvalue of A. IfA = 0, /(A) = <5 ^ 0; otherwise, 



/(A) 



e 8X 



A 



the numerator being nonzero for all A ^ if and only if the eigenvalue condition 
holds. I 



Example 3.4.2 Consider, as an example, the system E corresponding to the 
linearized pendulum (2.31), which was proved earlier to be controllable. In 
Appendix C.4 we compute 

tA J cos t sin t 
- sin t cos t 



Thus, for any <5 > 0, 



,-> i r, , / sin S 



1 1 cos o 



which has determinant (—sin 5). By Lemma 3.4.1, E is (5-sampled controllable 
iff 

sin (5^0 and 2kiri ^ ±i<5 , 

i.e., if and only if <5 is not a multiple of ir. 

Take, for instance, the sampling time <5 = 2ir. From the explicit form of e , 
we know that e = /. Thus, 

AW =A- 1 (e SA -I) = 0, 
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so G = 0. This means that the discrete-time system Em has the evolution 
equation 

x(t+l) = x{t). 

No matter what (constant) control is applied during the sampling interval [0, 5], 
the state (position and velocity) is the same at the end of the interval as it 
was at the start of the period. (Intuitively say for the linearized pendulum, we 
are acting against the natural motion for half the interval duration, and with 
the natural motion during the other half.) Consider now the case when 5 = ir, 
which according to the above Lemma should also result in noncontrollability of 
£ [<5] . Here 

F = e SA = -I 

and 

A {S) = A-\e SA - I) = -2A- 1 = 2A , 



so 

G=2AB 



Thus, the discrete-time system Em has the evolution equations: 

Xl {t+1) = -xi(i) + 2u(i) 

x 2 {t+l) = -x 2 {t). 

This means that we now can partially control the system, since the first coordi- 
nate (position) can be modified arbitrarily by applying suitable controls u. On 
the other hand, the value of the second coordinate (velocity) cannot be modi- 
fied in any way, and in fact at times 5, 26, . . . it will oscillate between the values 
±£2(0), independently of the (constant) control applied during the interval. □ 

Exercise 3.4.3 Consider the system (with U = R, X = R 2 ) 

Xl = X 2 

±2 = —Xl - X 2 + U, 

which models a linearized pendulum with damping. Find explicitly the systems 
Sm, for each 5. Characterize (without using the next Theorem) the S's for 
which the system is (5-sampled controllable. □ 

The example that we discussed above suggests that controllability will be 
preserved provided that we sample at a frequency 1/5 that is larger than twice 
the natural frequency (there, 1/2-ir) of the system. The next result, sometimes 
known as the "Kalman-Ho-Narendra" criterion, and the Lemma following it, 
make this precise. 
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Theorem 4 Let K = R and Zei E = (A, B) oe a controllable continuous-time 
(time-invariant) linear system. If 5 > is swc/i that 



S(\ - n) ± 2km, fc = ±l,±2, 



/or every two eigenvalues X, fi of A, then E is aZso 5-sampled controllable. 

Proof. We will prove that (e , B) is a controllable pair. The conclusion then 
will be a consequence of Lemma 3.4.1, because if A were to have any eigenvalue 
A of the form 2km/5, k = nonzero integer, then, with this A, 

S(X - A) = 4km , 

contradicting the hypothesis. 

We now establish that the Hautus criterion must hold for (e , B). By the 
Spectral Mapping Theorem, all eigenvalues of e are of the form e , where A 
is an eigenvalue of A. So we must check that 

Tmk[e SX I-e SA ,B] = n 

for each such A. Equivalently, we must prove that for each nonzero vector p 
which is an eigenvector of (e )', necessarily 

p'B^Q. (3.12) 

By controllability of (A, B) the conclusion (3.12) holds if p is an eigenvector 
of A'. So the only step left is to show that A' and (e )' = e have the 
same eigenvectors. Lemma A. 3. 3 in the Appendix on linear algebra implies 
this, provided that /(A) := e is one-to-one on the spectrum of A. But this is 
precisely what the hypothesis of the Theorem asserts. I 

The condition in the above Theorem is not necessary except when the num- 
ber of controls m = 1 (see below). A simpler, though stronger, sufficient con- 
dition is as follows. For any given system E = (A,B), a frequency o/E is any 
number of the form 

|ImA| 

where A is an eigenvalue of A. For instance, for the system in Example 3.4.2 
there is only one frequency, 1/27T. This is the frequency of oscillation of the 
solutions of the uncontrolled system x = Ax, which are all combinations of sin t 
and cost (hence of period 2tt). Note that B does not affect the frequencies. 

Lemma/Exercise 3.4.4 A controllable continuous-time time-invariant sys- 
tem E = (A, B) remains controllable provided that the sampling frequency 
1/(5 is larger than 2a> for every frequency to of E. □ 
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Remark 3.4.5 When m = 1, the eigenvalue condition stated in Theorem 4 
is also necessary. This is proved as follows: Assume that (A, B) is <5-sampled 
controllable but that A / \x are eigenvalues of A such that 

Choose two (necessarily linearly independent) eigenvectors: 

Av = Av, Aw = fj,w . 
Then also 

<5 A i ft A 

e v = av and e w = aw , 

from which it follows that a is an eigenvalue of e of geometric multiplicity 
greater than one, contradicting Lemma 3.3.9. □ 

Exercise 3.4.6 Give an example to show that the eigenvalue condition in The- 
orem 4 is not necessary (if m > 1). □ 

Remark 3.4.7 There is a somewhat different way of proving Theorem 4, which 
is more conceptual. We give it now. The main point is that under the hypotheses 
of the Theorem it can be proved that A is an analytic function of e , so that 
A is a linear combination of the matrices 

e k6A ,k = 0,...,n-l. 

Once this is known, it follows that each column of R(A, B) is a linear combi- 
nation of the columns of R(e , B), and hence the second matrix cannot have 
rank less than n, since the first one has rank n by the controllability hypothesis. 
It then follows that the pair (e , B) is controllable, as desired. 

We now show that A is an analytic function of e . (The proof requires the 
concept of analytic but nonentire functions of matrices.) Consider the exponen- 
tial function f(s) = e . The hypothesis means precisely that / is one-to-one 
on the set 

A = {Ai,...,Aj} 

of eigenvalues of A. Pick disjoint neighborhoods 

01,02,- ..,0J 

of 

/(Ai),/(A 2 ),...,/(A0 

respectively. Since / is everywhere nonsingular, we may apply the Inverse Func- 
tion Theorem to conclude that, taking smaller 0fe's if necessary, there exist 
disjoint neighborhoods 

0i,0 2 °,...,0° 

of 

Al, A2, • ■ • , A; 
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and diffeomorphisms 

9k : O k -» Ol 

such that 

9k ° Ik = identity on each Oj, . 

Finally let O (resp. O ) be the union of the sets Ok (resp. 0°), and let g be 
the extension of the <7fc's. Then g o / equals the identity in the open set O 
which contains the eigenvalues of A, and hence 

g(f(A)) = A, 

which proves that indeed A is a function of e = f{A). □ 

3.5 More on Linear Controllability 

This section develops some of the basic facts about controllability of linear sys- 
tems, possibly time-varying. The focus is on the continuous-time case; the 
discrete-time situation is in part analogous and far simpler (and left as an ex- 
ercise), although some aspects cannot be generalized. 

We start with a review of some basic properties of generalized inverses of 
operators. Appendix A. 2 reviews pseudoinverses of matrices; since here the 
interest is on the operator from controls to states, and controls belong to an 
infinite dimensional space, a somewhat more general result is needed. On the 
other hand, we restrict for simplicity of exposition solely to the case of onto 
operators. 

Pseudoinverses 

Let K = R or C. Recall that an inner product space H over IK is a vector space 
together with a binary operation (x, y) such that (x, jiJgK for all x,y € H and 
the following properties hold for all x,y, z G H and all «,/5gI: 



(a) (a;, y) = (y, x) (overbars denote complex conjugation; thus, in the case 
K = R, we are simply saying that inner products are symmetric); 

(b) (x, ay + (3z) = a(x,y) + @(x, z) ; and 

(c) (a;, x) > for x ^ and (0, 0) = 0. 

Given such an inner product, one defines an associated norm ||ar|| := \/(x, x) , 
and H becomes a metric space with the distance function d(x, y) := \\x — y\\. If 
H is complete as a metric space with this distance, then H is called a Hilbert 
space. 

There are two particular types of Hilbert spaces that we use. For any fixed 
n, we view IK" as the set of all column n- vectors with the inner product 



(x,y) := x*y = ^ 



■*' i,JJi 
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(Here "*" denotes conjugate transpose.) The other type of Hilbert space that 
we work with is infinite dimensional, and is defined as follows. For each positive 
integer m, we let £„(<7, r) be the set of all square integrable functions uj : 
[a, r) — > K m . Such an u) can be thought of as a column m- vector of square 
integrable functions. The set Cf n ( a > T ) ^ s a Hilbert space with the inner product 



(u,v) := / u>(t)*v(t) dt. (3.13) 

J a 

Let fi and X be two Hilbert spaces. Given a continuous linear mapping (in 
functional-analytic terms, a bounded operator) 

L: fl^X, 

there exists always an adjoint operator L* , defined by the property that 

(Llu , x) x = (w , L* x) n for all x G X, lu G f2 . 

For instance, if £7 = K m and X = K", and L is thought of as an n x m matrix 
(with respect to the canonical bases of K m ,K™) then L* is just the conjugate 
transpose of L. 

Example 3.5.1 As another example, consider the case £1 = C 2 m (a, r), X = K™, 
and 



Llu:= / k{t)*cu{t)dt, (3.14) 

J a 

where k is a fixed m x n matrix of elements of C 2 (o~, r). Thus, if we denote by 
ki the ith column of k, i = 1, . . . , n, then each hi is in C 2 m (p, r) and 

Lw= : 

We now compute the adjoint of L. Pick any ui G O and a; G X. Since (Lw, cc) 
equals 

k(t)*u)(t)dt) X=( u)(t)* k(t) dt) x = uj(t)*(k(t)x)dt, 

we have that L*x is the element of C 2 m given by the function (L*x)(t) = k{t)x. □ 

From now on we assume that X is finite dimensional. (Most of what we do 
generalizes easily to the case of arbitrary X but L of closed range. The latter 
property is automatic in the finite dimensional case.) 

The following Lemma is basic. Here "im" denotes image, "ker" kernel or 
nullspace, and "~ L " indicates orthogonal complement: 

S = {z\ (x, z) = for all i6S). 

Since X was assumed finite dimensional, it holds that (5^)^ = S for all sub- 
spaces S of X. 
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Lemma 3.5.2 For any L as above, imLL* = (kerL*)- 1 - = imL. 

Proof. It is clear that imLL* C imL. Since u± " reverses inclusions, it will be 
enough to prove that 

(imLL*)- 1 C kerL* C (imL)- 1 . 

Pick any z G (imLL*)- 1 -. Then, (LL*x, z) = for all x € X, so in particular for 
x = z: 

0= (LL*z,z) = (L*z,L*z) = \\L*z\\ 2 , 

which implies that z G kerL*. Hence also z € (imL)- 1 -, as desired, because for 
w e O, (Lu>,z) = (uj,L*z) = 0. I 

We let 

W :X^X, W := LL* . (3.15) 

When X = K", we identify W with an n x n matrix. Note that 

W* = (LL*)* = L**L* = LL* = W , (3.16) 

i.e., W is self- adjoint. (When X = K™, W is a Hermitian matrix.) Moreover, 
W is positive semidefinite, because 

(x,Wx) = \\L*xf 

for all x. Since X is finite dimensional, the following are equivalent: W is onto, 
one-to-one, invertible, positive definite. We can then conclude as follows. 

Corollary 3.5.3 The following statements are equivalent for L, W as above: 

(a) L is onto. 

(b) L* is one-to-one. 

(c) W is onto. 

(d) detl^/0. 

(e) W is positive definite. □ 
Consider again the situation in Example 3.5.1. Here L is onto iff the matrix 



W = k{t)*k{t) dt >0. (3.17) 

J a 

Equivalently, L is onto iff L* is one-to-one, i.e., 

there is no p ^ in X with k(t)p = for almost all t € [cr, r) , (3.18) 

or, with a slight rewrite and ki := ith column of k*: 

(p, ki) = for all i and almost all t => p = . (3.19) 



2 
~X\\ 
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Proposition 3.5.4 Assume that L is onto, and let 

L* := L^LL*)- 1 : X^ O. (3.20) 

Then, L#x is the unique solution of Luj = x of smallest possible norm, i.e., 

• L(L#x) = x for all x G X, and 

• L^a; < ||oj|| for every u for which Lcj = x and u> 7^ L^a;. 

Proof. Since 

LL* = (LU)(LVy 1 = identity , 

the first property is clear. Assume now that Llu = x. We will prove that 

(u> — L#x, L#x) = 0, so 

|M| 2 = ||^-L # x|| 2 +||L#x|| 2 

will give the desired conclusion. But 

(to — L#x, L#x) = (uj, L#x) — \\L#z 

so we prove that the two terms on the right are equal: 

(uj,L*x) = (u,L*(LL*y 1 x) = {Luj,{LL*y l x) = (x, {LL*y 1 x) 

= {LL*{LL*)- 1 x,{LL*y l x) = (L* (Li*)" 1 ^ L* {LL*y 1 x^ 
- \\t#t\\ 2 

This completes the proof. I 

The operator L# is the generalized inverse or (Moore-Penrose) pseudoinverse 
of L. When L is not onto, one may define an operator with the minimizing 
property in the above result restated in a least-squares sense; we omit the details 
here since only the case when L is onto will be needed (see Appendix A. 2 for 
the finite dimensional case). 

In Example 3.5.1, 

(L*x)(t) = k(t) 

Note that if the entries of k happen to be essentially bounded (not just square 
integrable), then L#x is also essentially bounded. 

It is easy to compute the operator norm of _L# in terms of the matrix W. 
We are defining the operator norm as 

||L # || := sup ||i # a;|| 

||x|| = l 



/ k(s)*k(s) ds 

Jo 


-1 

X 
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with respect to the C^ norm. From the definitions (3.15) and (3.20), it follows 
that, for each vector x, 

\\L*x\\ 2 = W^W^x]] 2 

= {L*W~ l x,L*W~ l x) 

= (LL*W- 1 x,W~ 1 x) 

= {x^W^x). 

Therefore, 

||L # || = ll^" 1 !! 172 ; (3.21) 

that is, the norm of the pseudoinverse is the square root of l/<7 m ; n , where cr m i n 
is the smallest eigenvalue of the positive definite matrix W. 

Application to Controllability 

Recall from Lemma 3.1.7(a) that the continuous-time linear system E is control- 
lable on [a, t] iff it holds that (0, a) -^ (y, r) for all t/6l. Thus, controllability 
means precisely that the following operator is onto: 



N : £™(<t,t) ^K™ : w i-> </>(r, a, 0, u>) = / $(r, s)B(s)u){s) ds . 

J a 

Thus, the extension of N to £^, with k(s) := B(s)*G>(t, s)*: 

L: C 2 m {a,r) -> K™ : cj ^ / fe(s)*w(s) ds , (3.22) 



is also onto. Note that entries of B are essentially bounded, by the definition of 
a continuous-time linear system, and that the entries of < 1 ) (t, s) are (absolutely) 
continuous in s. Thus, each entry of k is in C^(a, r) and hence also in £„(<7, t), 
so L is an operator as in Example 3.5.1. (When considering the extension L of 
N, we really are dealing with the system E "with C 2 controls" in the sense of 
Exercise 2.7.5, but we do not need to use this fact explicitly.) 

Given any E, we consider the operator L defined by formula (3.22). If E is 
controllable, L is onto. Conversely, assume that L is onto. Then for each iGl, 
ui := L#x is such that Lui = x. Since the entries of k are essentially bounded, 
it follows that ui is also essentially bounded. So uj is in the domain of N, and 
hence x is in the image of N. We conclude that L is onto iff N is, that is, iff 
E is controllable. (Alternatively, we could argue that L is onto iff N is, using 
the fact that L^ is dense in C^ and that both TV and L are continuous.) We 
apply the above results to this L, to obtain the following conclusion. For the 
last part, recall that 

4>{t, a, x, lu) = z iff z — (j){r, a, x, 0) = Nlu . 
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Theorem 5 Assume that E is a continuous-time linear system, a < r £ R, 
and denote by bi the ith column of B. The following statements are equivalent: 

1. E is controllable in [a, r]; 

2. W{<j,t) = SI ®(t,s)B(s)B(s)*${t,s)* ds > 0; 

3. There is no nonzero vector p £ X such that {p, < 1 ) (t, s)bi(s)) = for almost 
all s £ [a, t] and all i. 

If the above are satisfied, then, given any two x, z £ X, the control given by the 
formula 

u>(t) = B{t)*<t>{T, t)*W{a, t)-\z - $(r, a)x) (3.23) 

is the unique one that minimizes \\u)\\ among all those controls satisfying 

cj>(t,o~,x,u>) = z. □ 

Exercise 3.5.5 Let E be a continuous-time linear system, and pick a < t £ R. 
Consider the controllability Gramian 

W c (<j,t) := J ${<T,s)B{s)B{s)*${cr,s)* ds . 

J cr 

Show: E is controllable in [a, r] if and only if W c (o~, r) has rank n, and, in that 
case, the unique control of minimum square norm that steers x to is given by 
the formula u(t) = -B(t)*$>(a,t)*W c {o-,T)- 1 x. □ 

Example 3.5.6 Consider the harmonic oscillator (linearized pendulum) 

X\ = X 2 

X2 = ~X\ + U. 

Assume that we wish to find a control transferring 

1 



to z = in time 2ir while minimizing the "energy" 

r-2-rr 

u(tfdt. 



We apply the Theorem. First note that, using e as computed in Example 
3.4.2, there results 



W 



sm s — sin s cos s \ . 

ds 



n \ — sins cos s cos 2 s 



ttI. 



110 3. Reachability and Controllability 



so 



Therefore, 



W~\z - *(27r,0)a;) = (-1/oQ 



W 7T ^ V S1I1 ( 27r _ *) COs(2tT - t) ,/ ^0/ 7T 

gives the desired control. □ 

The above Theorem provides to some extent a computable criterion for con- 
trollability in [a, t] . One may obtain numerically the fundamental solution and 
then perform the required integration to get the constant matrix W(a, r), whose 
determinant is then checked to be nonzero. Below we develop a criterion, ap- 
plicable when the entries of A and B are smooth as functions of time, that 
does not require integration of the differential equation. In practice, of course, 
round-off errors will make the determinant of W(a,r) always nonzero. Thus, 
controllability cannot be decided precisely. Furthermore, if the determinant of 
W turns out to be small, even if the system is controllable, it is not so in a prac- 
tical sense, since the necessary controls will tend to be very large, as suggested 
by the term W~ l that appears in the expression of the minimum norm control. 
This is related to the fact that even though most systems are controllable (at 
least in the time-invariant case), many may be "close" to being uncontrollable; 
recall the discussion after the proof of Proposition 3.3.12. It is also related to 
the fact that trying to control in too short an interval will require large con- 
trols, no matter how controllable the system is (since the integral defining W is 
taken over a small interval). The next exercise deals with a particular case of 
an estimate of how large such controls must be. 



Exercise 3.5.7 For the system 








±i = 


= x 2 




X2 = 


= u 



show that the operator L# corresponding to a = 0, r = £ satisfies 

||l#|| =0(e-i) 

as e — > 0. (Hint: Use (3.21) by showing that the smallest eigenvalue of W(0,e) 

is of the form 

3 

12 v ; 
The power series expansion for yl + a may be useful here.) □ 

We may restate condition (3) in a slightly different form, which will be useful 
for our later study of observability. Transposing the conclusion (d) of Lemma 
C.4.1, one has the following: 
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Lemma 3.5.8 For any A(-), the fundamental solution ^(t, s) associated to the 
adjoint equation 

p(t) = -A{t)*p{t) (3.24) 

is *(M) : = $(«,*)*• □ 

Corollary 3.5.9 The continuous-time linear system £ is controllable on [a, r] 
if and only if there does not exist a nonzero solution pit) of the adjoint equation 
on [a, t] such that 

(p(t),bi(t)) = for almost all t £ [cr, r] 

for all i = 1, . . . , to (recall that 6j is the ith column of B). 

Proof. We show that the existence of such a solution would be equivalent to 
the property in (3) of Theorem 5 (p. 109). For this, note simply that if p is a 
solution of the adjoint equation on [a, r] then pit) = ^(t, t)p(t) = < 1 ) (t, t)*p(r), 
so 

(p(t),b) = ( P (T),<5>(T,t)b) 

for any vector b, and that p(t) is nonzero for all t if and only if p(r) ^ 0. I 

The last part of Theorem 5 provides a solution to a particular type of opti- 
mal control problem, that of minimizing the norm of the control that affects a 
given state transfer. In applications, this norm is typically proportional to some 
measure of the energy used in controlling the system. There is a variation of this 
problem that can be solved with precisely the same techniques. Assume that 
the coordinates of the control u are assigned different costs. For instance, we 
may want to consider a situation in which there are two independent controls 
and it is 10 times more expensive to apply a unit of the first control as it is 
of the second. In that case, it would make sense to try to minimize the cost 
(10ui) 2 + u\ = lOOw 2 + u\. More generally, we may have a cost of the type 

Exercise 3.5.10 Let S be a controllable continuous-time linear system, and let 
Q be a real symmetric positive definite m x m matrix. Pick any x, z € X, and 
a, t £ K. Find a formula for a control uj £ £^(er, r) which gives </>(r, a, x,lu) = z 
while minimizing 

(<T 

uj(s)*Qlo(s) ds . 

Prove that for each pair x, z there is a unique such control. Do this in two 
alternative ways: (1) Applying again the material about pseudoinverses, but 
using a different inner product in the set C^- (2) Factoring Q = QlQi and 
observing that the same result is obtained after a change of variables. □ 

Remark 3.5.11 Optimal control problems for linear systems with quadratic 
cost result typically in solutions that are linear in a suitable sense. For instance, 
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the optimal control in Theorem 5 (p. 109) is linear in both x and z. This is 
not surprising, since in an abstract sense minimizing a quadratic function (here, 
the norm of the control) subject to linear constraints (the linear differential 
equation) can be thought of, after elimination, as an unconstrained minimization 
of a quadratic function, which in turn can be solved by setting derivatives to zero; 
since derivatives of quadratic functions are linear, the solution to the original 
problem becomes linear in the original data. (This kind of reasoning can be 
made precise through the study of abstract quadratic optimization problems in 
infinite dimensional spaces.) The linearity property means that in general, linear 
systems problems with quadratic costs are far easier to solve than, say, linear 
problems with nonquadratic costs or nonlinear control problems with quadratic 
costs. Chapter 8 studies other linear-quadratic problems, in which the final 
state is not constrained and costs are imposed along trajectories. □ 

Remark 3.5.12 Theorem 5 (p. 109) shows that among all controls effecting the 
transfer (x, a) ~» (z, r) there is a unique one of minimum norm. Of course, if one 
does not impose the minimality requirement, there are many such controls, since 
N maps an infinite dimensional space into a finite dimensional one. Many dense 
subsets of £^ will map onto all of the state space. For instance, there are always 
in the controllable case piecewise constant controls so that (x, a) ~> (z, t). (See 
more about this in Section 3.9.) 

Observe also that the explicit formula for the minimum norm control shows 
that this control is of class C k provided that A(-) and B(-) are of class C k . □ 

Exercise 3.5.13 Let E be a discrete-time linear system with IX = K m ,X = 
K",K = 1 or C. State and prove the analogue of Theorem 5 (p. 109) for E. 
(Hint: Instead of <$>(<t,t) use now the matrix product 

A(r-l)A(T-2)...A(a). 

Let £1 := K mT , T = r — a, and apply the pseudoinverse results again.) □ 

Exercise 3.5.14 For this problem, we call a time-invariant system output con- 
trollable if it holds that for each i£l and each y G y there exists a T > and 
a control u such that h((j>(T,x,u)) = y. Prove that a time-invariant continu- 
ous-time linear system (A, B, C) is output controllable iff 

rank CR(A, B) = rank [CB, ..., CA n ~ 1 B\ = p. □ 

Exercise 3.5.15 (Optional, computer-based.) In Exercise 3.2.12, calculate nu- 
merically, using the pseudoinverse formula, controls of minimal energy transfer- 
ring one given state to another in time T = 1. (Unrealistically, but for mathe- 
matical simplicity, take all constants equal to one.) □ 

A Rank Condition 

The characterization of controllability given in Theorem 5 involves integration 
of the differential equation of the system. We now look for a simpler condition, 
analogous to the Kalman rank condition for time-invariant systems. 
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If E is a continuous-time linear system and X is an interval in R, we say 
that E is smoothly-varying on X iff A(t) and Bit) are smooth (infinitely 
differentiable) as functions of t for t € X, and that E is analytically varying 
on X iff A(t) and Bit) also are analytic on X. Analyticity means that for 
each t € X each entry of A and B can be expanded in a power series about t, 
convergent in some nontrivial interval (t — e, t + e) (see Appendix C). 

The matrix function $(s,£) is smooth (respectively, analytic,) as a function 
of t € X and sGl when E is smoothly (respectively, analytically) varying on X. 
Fix E and X so that E is smoothly varying on X, and fix a nontrivial subinterval 
[C) T ] Q 3-. Introduce the (smooth) n x m matrix function 

M (t):= *(T,t)B(t), 

and let 

M k {t) := {d k M /dt k ){t), k > 1 . 

Consider also the matrix of functions obtained by listing all the columns of the 
Mi,i = 0,...,k: 

M^(t):=(M (t),M 1 (t),...,M k (t)). 

Proposition 3.5.16 Let E be a continuous-time linear system. 

1. Assume that E is smoothly varying on X, and pick any a < r so that 
[C) t] C X. If there exists a to £ [it, t] and a nonnegative integer k such 
that 

rankM (fe) (t ) =n, 

then E is controllable. 

2. Assume now that E is analytically varying on X, and let to be any fixed 
element of X. Then, E is controllable on every nontrivial subinterval of X 
if and only if 

rankM (fc) (i ) = n 

for some integer k. 

Proof. If E were not controllable on [a, r], there would exist by condition (3) 
in Theorem 5 (p. 109) a nonzero vector p such that p*M (t) = for almost 
all t G [a, t], and hence by continuity of Mq, for all such t. It follows that all 
derivatives are also identically zero, that is, 

P *M k {t) = 

for all t € [a, r] and all k > 0. Thus, the rows of each matrix M^ k \t) are 
linearly dependent, so rankM^ 'it) < n for all t € [o'jT']. This proves the first 
statement. 
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We now prove sufficiency in the analytic case. If E were not controllable on 
[a, t], then, as in the smooth case, p*M k (t) = on [a, r], for all k. But then, 
by the principle of analytic continuation, also 

p*M k (t) = for alii el. 

Thus, if the rank is n for some t e X and some k, we obtain a contradiction. 

Conversely, suppose now that E is controllable and analytically varying on 
X. Assume that there is some to € X such that 

rankM( fe) Oo) < n 

for all k. For this to, let 

P k := { P eK n \p*M^(to) = 0}. 

By assumption, the subspaces P k are all nonzero. Since 

P 2 Pi 2 • • • , 

the dimensions of the P k are nonincreasing. Let Pr- have minimal dimension; 
then 

P k = P K for k > K . 

Pick p e P K - It follows that p*M k (t Q ) = for all k. By analyticity, p*M {t) = 
for t in an interval about to (expand in a power series about to), and so, by 
analytic continuation, this holds on all of X and in particular on all of [a, r]. 
This would contradict the assumed controllability of E. I 

The above result can be restated somewhat more elegantly in terms of the 
infinite matrix 

M(t):=(M (t),M 1 (t),...,M k (t),...). 

The system E is controllable if M(t ) has rank n for some to, and in the analytic 
case this condition is also necessary for every to- 

The condition would seem still to involve integration, since the transition 
matrix $ is used in defining the M k s. But we can obtain an equivalent condition 
as follows: Let Bo := B and, for each i > 1, 

B l+1 (t) := A{t)Bi{t) - -Bi{t) . 
Note that the -B, can be obtained directly from the data (A, B). 
Lemma 3.5.17 For all i and all t € X, M t {t) = {-l) l $>{T,t)B l {t). 
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Proof. By induction. The case i = holds by definition. Assume the Lemma 
proved for i, and consider Mj + i. Then, 

M i+ i(t) = jM l (t) = (-l) l j t MT,t)B l (t)] 



(-1)' 



It*™ 



Bi(t) + *(T,t)^-Bi(t) 

at 



= (-l) 4 {-<l>(T,tM(t)B J (t) + <l»(T,t)|s j (t) 

where the last equality holds by Lemma 0.4.1(d). But the last term is equal to 
(-l) i + l <t>( T ,t)B l+1 {t), as desired. I 

Since < &(t, i) is invertible for all r, t, p*Mj(i ) = if and only if q* Bi(t) = 0, 
with q = 3>(t, i)*p. We conclude: 

Corollary 3.5.18 The conclusions of Proposition 3.5.16 hold when 

rank [B (t ), Bi(i ), . . . , B fc (t )] = n (3.25) 

holds instead of the rank condition on M^ k '(to). □ 

Remark 3.5.19 In Proposition 3.5.16 and Corollary 3.5.18, infinite differen- 
tiability is not needed for the necessity statements, in the following sense: If 
A(-) and B(-) are fc-times differentiable and if M^ k '{to) or (3.25) have rank n 
at some to, then controllability already follows by the same argument. □ 

Exercise 3.5.20 Show by counterexample that without analyticity the rank 
condition is not necessary. More precisely, give two examples as follows: 

(i) A smooth system with n = m = 1 that is controllable in some nontrivial 
interval but for which there is some to so that M^ '(to) = for all k; and 

(ii) A smooth system with n = m = 2 that is controllable in some nontrivial 
interval but for which the rank is one for every to and k. □ 

Example 3.5.21 Consider the system with K = R, n = 3, m = 1, and matrices 

ft 1 0\ 

A(t) = t 3 B(t) = 




This system is smoothly (in fact, analytically) varying on (—00,00). Since 

/0 1 -l' 
[Bo(0),B 1 {0),B 2 (0),B 3 (0)]=\l 

\1 2 

and this matrix has rank 3, the system is controllable on every nontrivial interval 

krl. □ 
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Remark 3.5.22 In the time-invariant case, Bi(t) = A l ~ 1 B, so the above con- 
dition says that E is controllable on any nontrivial interval [<r, r] if and only 
if the columns of (B,AB,A 2 B, . . .) span an n-dimensional space, which as re- 
marked in Lemma 3.3.2 is equivalent to the condition rankR(A, B) = n. This 
provides yet another proof of Theorem 3 (p. 89). □ 

In general, for any fixed to, it may be necessary to check the rank of the 
matrix in Corollary 3.5.18 for arbitrarily large k. As an illustration, consider 
the analytic system 

x = t k u, 

where k is any integer. This is controllable on any nontrivial interval (just check 
rank condition at 1, for instance), but with to = we would have to check up 
to the fcth matrix. For generic to, k = n is enough, however, as shown in the 
following problem. (So, even though there is no "Cayley-Hamilton Theorem" 
to apply, as was the case for time-invariant systems, the situation is still good 
enough for almost every to.) 

Exercise 3.5.23 Let £ be a continuous-time linear system, analytically varying 
on X. Prove that if X is controllable on any nontrivial subinterval [a, r] then 

rmk[Bo(t),B 1 (t),...,B n _ 1 (t)]=n 

for almost all t G I. (Hint: First prove that if rankM^^i) = rankM (fc+1 )(£) 
for t in an open interval JC1, then there must exist another subinterval J'C J 
and analytic matrix functions 

V (t),...,V k (t) 

on J' such that 

k 

M k+1 {t) = Y J M l {t)V t {t) 

on J'. Conclude that then rankM^^) = rankM^(i) for all Z > k on J'. 
Argue now in terms of the sequence n k ■= maxjrankM^^i), t 6l}.) □ 

Exercise 3.5.24 Consider the continuous-time linear system over IK = M with 
n = 2, m = 1, and matrices 

Prove that the system is not controllable over any fixed interval, but that for 
every fixed <r in K, the ("frozen") time-invariant linear system (A{a), B(o~)) is 
controllable. Show noncontrollability of the time- varying system in two different 
ways: 

(i) Via the rank test; and 
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(ii) Alternatively by explicitly calculating solutions and noticing that the 
reachable set from the origin at any fixed instant is a line. □ 

Exercise 3.5.25 Refer to Exercise 3.2.12. Assume here that the ground speed 
is not constant but is instead a function of time, c = c(t) (take c to be smooth, 
though much less is needed). Apply the rank criteria for time- varying linear 
systems to conclude that the system is controllable in any nontrivial interval in 
which c is not identically zero. □ 

3.6 Bounded Controls* 

In this section, we deal with continuous-time time-invariant systems of the form 
X = Ax + Bu for which IX is a subset of W n . Linear systems have, by definition, 
U = R m , so we call such systems "linear systems with constrained controls". 
For instance, take the system x = —x + u (n = m = 1), with II = (— 1, 1). The 
pair (A, B) = (1,1) is controllable, but the system with restricted controls is 
not, since it is impossible to transfer the state x = to z = 2 (since x(t) < 
whenever x(t) G (1,2)). 

In order to avoid confusion with reachability for the associated linear systems 
with unconstrained controls, in this section we will say that z G K™ can be II- 
reached from x G R n (or that x can be II- controlled to z) in time T if there is 
some input uj : [0,T] — > K m so that 4>(T,0,x,uj) = z and uj(t) G II for (almost) 
all t G [0, T]. We define the reachable set in time T > 

1Z u {x) := {z G K" U-reachable from x in time T} 

and 1Zu{x) '■= Ut>o ^u( x )- (Thus 1Z-r« i {%) is the same as what we earlier called 
K(x).) 

In this section we will establish the following result. 

Theorem 6 Let M be a bounded neighborhood of zero. Then, 7?-ix(0) = R™ if 
and only if 

(a) the pair (A, B) is controllable, and 

(b) the matrix A has no eigenvalues with negative real part. 

Observe that the necessity of controllability for the pair [A, B) is obvious, 
since 7e u (0) Cft(0). 

We prove Theorem 6 after a series of preliminary results. 

Lemma 3.6.1 Let II C R m and pick any two S,T > 0. Then 

nZ(o) + e TA nUo) = n s u + T (o). 



* This section can be skipped with no loss of continuity. 
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Proof. 


Pick 
















Xl = 


Jo 


[T- 


-^Buj, 


i(r)dr: 


r S+T 
= / e( S+T - 

Js 


and 
























X2 = 


Jo 


(S-r)/ 


[ Buj 2 {T)dT 



with the inputs u>i It-valued. Note that 



e TA x 2 



e (S+T-r)A Buj2{T)dl 







Thus 



where 



rS + T 



/ W2(r) 0<t<S 

UJ[S) ~ \ wi(t-S) S<t<S + T. 

Note that w(i) G U for all £ G [0, S + T\. Thus ft£(0) + e TA K^{0) C ^ +T (0). 
The converse inclusion follows by reversing these steps. I 

By induction on q we then conclude: 
Corollary 3.6.2 Let II C M. m and pick any T > and any integer q > 1. Then 
7^(0) + e T ^S(0) + . . . + e^ TA Kl{0) = <(0) . 

Proposition 3.6.3 (1) If U C R™ is convex, then 7\Lxx(0) i s a convex subset of 

1". (2) If {A, B) is controllable and It C R m is a neighborhood of G K m , then 
7\Lu(0) is an open subset of 1". 

Proof. Convexity of each 7\^(0) follows from linearity of <j>(T, 0, 0, u) on w and 
convexity of IX. This proves that the (increasing) union TZu{0) is convex, when 
II is convex. 

Assume now that (A, B) is controllable and II C R m is a neighborhood of 
0. We first prove that, for each T > 0, 7^(0) is a neighborhood of G K™. 
Fix such a T. Pick a subset Uo C U which is a convex neighborhood of 0. 
The desired conclusion will follow if we show that G K™ is in the interior of 
TZjj^ (0), since IZ^ (0) C 7^(0). So without loss of generality, for the rest of this 
paragraph we replace II by Uo and hence assume that 11 is also convex. Pick 
any basis e\, . . . , e n of R n . Let e := — 5^™=i e «- ^or eacn * = 0, . . . ,n there is 
an input w%, not necessarily II- valued, so that 

ei = <f>(T,0,0,LJi) . 
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Let n > be so that, with u[ := j^Ui, i = 0, . . . , n, it holds for all i that w^(t) G IX 
for (almost) all £ G [0, T]. (There exists some such \x because U is a neighborhood 
of and the u>% are essentially bounded.) Thus e\ := -ei = <j>(T, 0, 0, lo'A G 1ZJ^(0) 
for each i. 

Pick any ei, . . . , e n such that \si\ < 2 fa+i~i for all i. Then 

£\ e n v-^ / 1 — e A/ 1 — £ / 

— d + . . . + — e„ = > — — +e i )e i + — — -e 

with e = 5^i £ i- This is a convex combination. Since all e\ G 7£.y(0), and this 
set is convex, it follows that —e\ + . . . + — e n G 7^.^(0) for all small enough 
ei, ... , e n , which shows that 7^.^(0) is a neighborhood of 0. 

Finally, we show that 7^u(0) is open. Pick any S > 0. By the previous 
discussion, there is some open subset V C 7£.S(0) containing zero. Pick any 
x G 7^u(0); we wish to show that some neighborhood of x is included in 7^u(0). 
Let u : [0,T] -» U be so that x = </>(T, 0,0, w). The set PF := e TA F is open, 
because e is nonsingular. For each y = e TA v G W, 

y + x = e TA v + <f>(T, 0, 0, u) G n^{V) . 

Thus a; + W is an open subset of TZ^(V) C 7?- u (0) C 7^u(0) which contains 
x. I 

For each eigenvalue A of A and each positive integer k we let 

J k , x := ker (A/ - A) k 

(a subspace of C") and the set of real parts 

j£ A := Re(J fe , A ) = {Rev | w G J fe , A } 

(a subspace of K"). Observe that if v G Jk.x, v = V1+W2 with Vj G 1", j = 1, 2, 
then «! G Jjf A , by definition, but also the imaginary part w 2 G J* A , because 
(— w) belongs to the subspace Jfc, A . We also let Jo, A = J^f A = {0}. 

Let L be the sum of the various spaces Jj? A , with Re A > 0, and let M 
be the sum of the various spaces Jj? A , with Re A < 0. Each of these spaces 
is A-invariant, because if v is an eigenvector of A, and v = V\ + ivi is its 
decomposition into real and imaginary parts, then the subspace of K™ spanned 
by v\ and Vi is A- invariant. From the Jordan form decomposition, we know 
that every element in C" can be written as a sum of elements in the various 
"generalized eigenspaces" Jk,\, so taking real parts we know that l n splits into 
the direct sum of L and M. (In fact, L is the largest invariant subspace on 
which all eigenvalues of A have nonnegative real parts, and analogously for M.) 

We will need this general observation: 

Lemma 3.6.4 If C is an open convex subset of R™ and L is a subspace of R™ 
contained in C, then C + L = C. 
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Proof. Clearly C = C + C C + L, so we only need prove the other inclusion. 
Pick any x G C and y G L. Then, for all s ^ 0: 



( 1 \ r/ s , / £ \ r/i 

a;+y= (.rr^J [ (1+£)a;] + 

Since C is open, (1 + e)x G C for some sufficiently small e > 0. Since L is a 
subspace, (^-^) y £ L C C. Thus x + y € C, by convexity. I 

The main technical fact needed is as follows. Fix any eigenvalue A = a + i(3 
of A with real part a > 0, and denote for simplicity J^ := J* y 

Lemma 3.6.5 Assume that (A, B) is controllable and IX C R m is a neighbor- 
hood of 0. Then jf C 7e u (0) for all fc. 

Proof. First replacing if necessary II by a convex subset, we may assume 
without loss of generality that 11 is a convex neighborhood of 0. We prove the 
statement by induction on k, the case k = being trivial. So assume that 
J*_i C lZu{0), and take any v G Jfe,A, v = wi + i«2- We must show that 
vi e K u {0). 

First pick any T > so that e ATj = e aT i for all j = 0, 1, . . .. (If j3 = one 
may take any T > 0; otherwise, we may use for instance T = fm-) Next choose 
any 5 > with the property that V\ := Sv\ G 72-^(0). (There is such a 5 because 
lZu(0) contains in its interior, by Proposition 3.6.3.) Since v G ker (XI — A) k , 
where v = 5v, 

e (A-\i)t v =h + t(A -XI)+-{A- XI) 2 + ..Av = v + w V* , 

where w G Jk-1- Thus 

e a *« = e A *w = e' A w - e xt w = e tA v - e at w Vi = jT, j = 0, 1, . . . . (3.26) 

Decomposing into real and imaginary parts w = w\ + iw2 and taking real parts 
in Equation (3.26), 

e at Vl = e tA Vl - e at Wl \ft = jT,j = 0, 1, . . . . 

Now pick any integer q > 1/5. Then 

9-1 
= Y^ e jTA vi + w' 

3=0 

where w' = — J^ e a ^ T W\ belongs to the subspace J^_ 1 - Applying first Corol- 
lary 3.6.2 and then Lemma 3.6.4, we conclude that 




pvi G 7^(0) + ^C K u (0) 
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where 



q-l qr-l 

P = 

j=0 j=o 



£^ T >£i = 5 >i 



(Here is precisely where we used that a > 0.) Therefore 5pv\ = pv\ G i?ix(0). 
On the other hand, 5p > 1 means that 

vi = -^ojwi + I 1 - ~r ) 
op \ op J 

is a convex combination. Since 5pv\ and both belong to 7\Lu(0), we conclude 
by convexity of the latter that indeed v\ G 7^u(0). ■ 

Corollary 3.6.6 Assume that (A, B) is controllable and IX C R m is a neigh- 
borhood of 0. Then L C 7e u (0). 

Proof. As before, we may assume without loss of generality that IX is convex. 
We have that L is the sum of the spaces Jj?^, over all eigenvalues A with real 
part nonnegative, and each of these spaces is included in 7\Lu(0). In general, if 
L\ and L^ are two subspaces of a convex set C, L\ + L^ C C (since x + y = 
2(22:) + 2-(2y)), so the sum of the L's is indeed included in lZu{0)- ■ 

The next result says that the reachable set from zero is a "thickened linear 
subspace" : 

Corollary 3.6.7 Assume that (A, B) is controllable and IX C R m is a convex 
and bounded neighborhood of 0. Then there exists a set B such that 7\Lu(0) = 
B + L and B is bounded, convex, and open relative to M. 

Proof. We claim that 7^ii(0) = (7^u(0) f]M) + L. One inclusion is clear from 
(ft u (0)f| M )+ L = K u (0) + L = K u (0) 

(applying Lemma 3.6.4). Conversely, any v G 7\Lu(0) can be decomposed as 
v = x + y G M + L; we need to show that x G Ru(0). But x = v — y G 
i?u(0) + L = -Rix(O) (applying the same Lemma yet again). This establishes the 
claim. 

We let B := 7\Lu(0) P| M . This set is convex and open in M because 7\Lu(0) 
is open and convex. We only need to prove that it is bounded. 

Let P : W 1 — > W 1 be the projection on M along L, that is, P(x + y) = x 
if x G M, y G L. Observe that PA = AP because each of L and M are A- 
invariant (so v = x + y, Ax G M, Ay G L, imply PAv = Ax = APv). Pick any 
x G TZu{0) f]M. Since x G TZu{0), there are some T and some u> so that 

e^ T - T ^ A Buo{T) dr . 
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On the other hand, since x £ M, x = Px. Thus: 

r T r T 

x = Px= / Pe {T - T)A Buj(T)dT= / e {T " T)A x(T) dr , 
Jo Jo 

where x{t) = PBoj{t) e Mf]PB(U) for all r. 

Since the restriction of A to M has all its eigenvalues with negative real part, 
there are positive constants c, fi > such that e* x\\ < ce _M * ||x|| for all t > 
and all x € M. Since PB(U) is bounded, there is then some constant d such 
that, if x is also in PB(U), \\e tA x\\ < c'e _Alt for all t > 0. So, for a; as above we 
conclude 

||a;|| < c' / e -^ T " r ) dr < -(1 - e^ T ) < - , 
Jo A 4 A* 

and we proved that £? is bounded. I 



Proof of Theorem 6 

Assume first that 1Z\i(0) = M n . We already remarked that the pair (A, B) must 
be reachable. If the eigenvalue condition (b) does not hold, then L is a proper 
subspace of R™ and M ^ 0. Enlarging U if necessary, we may assume that IX is 
convex and bounded. Lemma 3.6.7 claims that R n = 7\Lu(0) is then a subset of 
L + B, with bounded B, a contradiction. Conversely, assume that (a) and (b) 
hold. By Corollary 3.6.6, R n = L C K u (0). I 

We also say that the system S is II- controllable iiTZu{ x ) = R" for all x G M. n . 

Exercise 3.6.8 Let II be a bounded neighborhood of zero. Show: 

1. Every state can be tl-controlled to zero if and only if (a) the pair (A, B) 
is controllable and (b) the matrix A has no eigenvalues with positive real 
part. 

2. E is completely U-controllable (that is, TZ u {x) = E" for all x € R n ) if and 
only if (a) the pair (A, B) is controllable, and (b) all eigenvalues of the 
matrix A are purely imaginary. I 

3.7 First-Order Local Controllability 

For nonlinear systems, the best one often can do regarding controllability notions 
is to characterize local controllability — and even that problem is not totally 
understood. In order to talk about local notions, a topology must be introduced 
in the state space. Even though the only nontrivial results will be proved for 
discrete-time and continuous-time systems of class C , it is more natural to 
provide the basic definitions in somewhat more generality. A class called here 
topological systems is introduced, which captures the basic continuity properties 
needed. In Chapter 5, a few basic facts about stability are proved in the general 
context of such systems. From now on in this Chapter, and unless otherwise 
stated, system means topological system in the sense of Definition 3.7.1 below. 



3.7. First-Order Local Controllability 123 

Topological Systems 

If X is a metric space, we use d(a, b) to denote the distance between elements of 
X, and as earlier, we let d^ be the uniform distance between two time functions 
into X; that is, if 71, 72 : X —> X, for some interval 1CT, then 

^00(71,72) : =sup{d(7i(i), 72(f)), i el} 

(essential supremum when dealing with measurable functions and T = R) . We 
also use the notation 

Bp{%) := {2/ e X I d(x,y) < p} 
for the open ball of radius p centered at x, and 

Bpi x ) := {y e X I d(x,y) < p} 

for its closure. 

Definition 3.7.1 A topological system E is an object 

(T,X,IL» 

such that X is a metric space; E is a system when X is thought of just as a set; 
and for each a < t in T and each uj £ lv-' J ' T > , 

has an open domain and is continuous there as a map into X^ a ' T > (with metric 
doo). □ 

In other words, the following property must hold for each pair a < r in T: 
If u> is admissible for the state x, and if 



in X, then there is some integer TV such that lu is admissible for x n for all n > N , 
and 

lim d(4>(t,a,x n ,oj),(j)(t,a,x,u))) = 

n — *oo 

uniformly on t € [a, r] . 

From Theorem 1 (p. 57) and Lemma 2.8.3, the following is then immediate: 

Proposition 3.7.2 If E is either a continuous-time system, or a discrete-time 
system of class C , then E is topological when X is given the induced topology 
ofR™. □ 
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Remark 3.7.3 If £ is a topological system, then the map 4>(t, a, x, lu) is jointly 
continuous on (t, x), provided only that (j>(-,o~, x, to) be continuous for each fixed 
o~,x,u) (as happens with continuous-time systems). Indeed, given any o~,t,U), 
any £ for which u> is admissible, any t € [a, r], and any number e > 0, there is 
always, because of uniform continuity of (f> in x, a 5 > so that 

d(<j>(t, a, x, u)),(/)(t, a, y, u))) < e/2 

whenever d(x,y) < S. Furthermore, continuity of </>(•, a, x, u>) ensures that there 
is a 5' > so that 

d((j)(s,<j, x, u)),4>(t,<T, x,u>)) < e/2 

whenever s € [cr, r] is such that \t — s\ < 6' . Therefore, also 

d(<f>(s, cr, x, u)),(/)(t, a, y, u))) < e 

if (s,x) is close to (t,y), as required. □ 

An equilibrium state x° is one for which there exists a control value u° G It 
such that 

(f>(T,<7,X ,UJ )=X° 

for each a < r, cr, r G T, where cj is the control with cj°(i) = u°. When the 
system S is linear, we always take x a = and w° = 0. Unless otherwise stated, 
x° is a fixed but arbitrary equilibrium state. 

Definition 3.7.4 Let £ be a topological system. Let £ be any path on an interval 
X = [c,t], and denote 

xa ■= £{o),xi := £(t) . 

The system Y, is locally controllable along £ if for each e > there is some 
5 > such that the following property holds: For each z,y G X with d(z, xq) < S 
and d(y,xi) < 6 there is some path C on X such that 

C(er) = z, C(t) = V 

and 

doo(C0 < £ - 

When x is an equilibrium state and £ = x on the interval [cr, r] , T = t — a, 
one simply says that Y, is locally controllable (in time T) at x. □ 

Thus, local controllability along a trajectory corresponds to the possibility 
of controlling every initial state near the original initial state to every final state 
near the original final state, and being able to do so without deviating far from 
the original trajectory. For equilibrium states, the initial and final states are 
the same, and the definition can be put in slightly different terms, using the 
following terminology: 
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Figure 3.2: Local controllability along trajectory. 



Definition 3.7.5 Let E be a topological system, and let y,z be in X. IfV is a 
subset ofX containing y and z, we say that z can be controlled to y without 

leaving V [in time T] (or, that y can be reached from z inside V) if there exists 
some path £ on some interval [a, r] [respectively, with T = r — a] such that 



and also 



for all t € [a, t] . 



C{cr) = z and ((t) = y 

C(t) e v 



□ 



Remark 3.7.6 Using the above definition, one can restate: The system E is 
locally controllable in time T at the equilibrium state x if and only if for every 
neighborhood V of x there is another neighborhood W of x such that, for any 
pair of elements z,y in W, z can be controlled to y inside V in time T (see 
Figure 3.3). □ 

For linear systems, local controllability is not an interesting notion: 

Lemma/Exercise 3.7.7 For any discrete-time or continuous-time linear sys- 
tem E and each pair a < t in T, the following properties are equivalent: 

(a) E is locally controllable along some trajectory T on [a, r]. 

(b) E is locally controllable along every trajectory T on [it, t]. 

(c) E is controllable on [a, r]. □ 

The following is an example of the "linearization principle" mentioned in 
Chapter 1. In particular, for continuous-time systems x = f(x, u), with (0, 0) € 
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Figure 3.3: Local controllability at a state. 



IxU and /(0, 0) = 0, the result asserts that controllability of the pair (A, B) 
obtained by expanding to first order x = Ax + Bu + o(x, u) is sufficient to insure 
local controllability about x = 0. 

Theorem 7 Assume that E is a continuous-time system of class C . Then, 

1. Let r = (£, u>) be a trajectory for E on an interval X = [a, r]. Then, a 
sufficient condition for E to be locally controllable along £ is that E* [r] be 
controllable on [a, r] . 

2. Let E &e time-invariant, and assume that x is an equilibrium state. Pick 
any e > and any !i£ll such that f(x, u) = 0. Then, a sufficient condi- 
tion for E to be locally controllable at x in time e is that the linearization 
of E at (x, u) be controllable. 

Proof. The second part is a special case of the first, so we prove part (1). Let 
Xq := £(<r) and X\ := £( T )- Consider the mapping 



a : V„ 



X, a(x, v) := 4>(t, a, x, v) . 



Note that a(xo,oj) = X\. By Theorem 1 (p. 57), Part 3, this mapping is contin- 
uously differentiable, and its partial differential with respect to v evaluated at 
(xq, lu) is the mapping 4>t{t, <t, 0, •), which is onto by the controllability assump- 
tion. It follows using the Implicit Mapping Theorem for maps from a normed 
space into a finite dimensional space, as in Theorem 53 (p. 465) in Appendix B, 
that there exists an E\ > and a continuously differentiable mapping 



j :B Sl (x ) x8 Cl (n) -» C; 



such that 



a(z,j{z,y)) = y 

for all (z,y) in the domain of j and such that j(xo,Xi) = u>. Given now any 
e > 0, pick 5 > small enough so that 
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whenever z £ Bg(xo) and y £ Bg(xi); such a 5 exists by continuity of j and of 
tp, and it is as desired for local controllability. I 

Lemma/Exercise 3.7.8 For discrete-time systems of class C , the same con- 
clusions as in Theorem 7 hold. □ 

Remark 3.7.9 The definition of local controllability can be weakened in many 
ways while still preserving the idea that every state close to the original state 
can be controlled to every state close to the final one. One such weakening is to 
allow for large excursions. For instance, consider the following bilinear system 
inR 2 , with IX = R: 

-l\ (1 

i o ) +u {o 1 

or in polar coordinates away from the origin, 

e = i 

p = up. 

Pick any nonzero state, say, x := (1, 0)'. From the polar coordinate description, 
it is clear that the system restricted to R 2 — {0} is controllable and, in particular, 
that every state near x can be controlled to every other state near x. However, 
the only way to control x to (1 + e, 0)', for e small, is through a motion of time 
at least 2n that circles about the origin, and in particular a motion that leaves 
the ball of radius 1 about x. Therefore, this system is not locally controllable 
in our sense. □ 

The following are some examples of locally controllable systems and the 
application of the linearization principle. 

Example 3.7.10 With X = R 2 and U = R, consider the system 

x\ = Xi + sin £2 + x\e X2 

±2 = x\ + U. 

This is locally controllable at x = 0, because the linearization at (0,0), 

satisfies the controllability rank condition. When u 2 is substituted for u in the 
second equation, the linear test fails, since now B = 0. In fact, the system is 
not locally controllable there, since starting at it is impossible to reach any 
states of the form ( ) with x^ < 0. On the other hand, the linear test is not 
definitive: For instance, if we have u 3 instead of u in the second equation, then 
the linear test fails (again B = 0) but the system is still locally controllable. 
This last fact follows from the observation that, given any trajectory (£,w) of 
the original system, then (£, v) is a trajectory of the system that has u 3 in the 
second equation, if we define v(t) := w(t) 1 ' 3 . □ 
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Example 3.7.11 Consider the system with X = IX = R and equations 

x = 1 + u sin x . 

We claim that this system is controllable along every trajectory T = (£, w) with 
to smooth defined on any X = [ct,t] (with a < r). Indeed, the linearization 
matrices (here lxl) are 

A(t) = (w(i)sin2£(i)) B{t) = (sin 2 £(t)) . 

So rank_B(to) = 1 for at least one to £ [ (J i T \ if and only if sin£(£) is not 
identically equal to zero. But if the latter happens, then 

which implies that £ cannot be constant. 

The assumption that u) is smooth is not necessary, as it is not hard to 
prove controllability of the linearization directly, but this assumption serves to 
illustrate the use of the controllability criteria for linear time- varying systems. 
In this example, it is not hard to establish global controllability of the original 
system. □ 

Exercise 3.7.12 Prove that, for the above example, 

rank(B (t),-Bi(t),B 2 (t)) = l 

for all t. □ 



3.8 Controllability of Recurrent Nets* 

We turn to studying controllability of what are often called "recurrent neural 
networks" . These constitute a class of nonlinear systems which, although for- 
mally analogous to linear systems, exhibit interesting nonlinear characteristics, 
and arise often in applications, and for which strong conclusions can be derived. 
In this section, we use 6 to denote the hyperbolic tangent function 

6(x) = tanhx : R -> R : ih 6 . 



This is sometimes called the "sigmoid" or "logistic" map; see Figure 3.4. 
For each positive integer n, we let 6 n denote the diagonal mapping 

•si\ fO( Xl )\ 

: . (3.27) 

B{x n ) 



9 n : R n -> R n 



* This section can be skipped with no loss of continuity. 
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Figure 3.4: tanh 

Definition 3.8.1 An n- dimensional, ma-input recurrent net is a continuous- 
time time-invariant system with X = K™, IX = W 71 , and equations of the form, 



x{t) = e n {Ax{t) + Bu{t)) , 
where A e M. nxn and B e R nxm . 



(3.28) 

D 



Since 9 n is a globally Lipschitz map, such a system is complete (every input 
is admissible for every state). Observe that, if in place of 9 = tanh we had 
9 = the identity function, we would be studying continuous-time time-invariant 
linear systems. Recurrent nets arise when the rates of change of state variables 
are bounded (\ii\ < 1), and one models this limitation by means of 9 n , which 
provides a smooth distortion, or saturation, of the rates of change. In system- 
theoretic terms, we may represent a net by means of a block diagram as in 
Figure 3.5. 



B 



?-* 


On 


►■ 


I 
















A 













Figure 3.5: Block diagram of recurrent net. 



We need the following concept, which we define for more general systems. 

Definition 3.8.2 A system £, with state-space X C W l , is strongly locally 
controllable around x° € X if for each neighborhood V of x° there is some 
neighborhood W of x , included in V, so that, for every pair of states y and z 
in W , y can be controlled to z without leaving V. □ 

Related notions are studied in Section 3.7; intuitively, we are asking that 
any two states that are sufficiently close to x° can be steered to one another 
without large excursions. 

For each pair of positive integers n and m, we let 



B, 



{Be 



Vi, roWi(B) j^ and Vi 7^ j, roWi(B) 7^ ±roWj(B)\ 



where rowj(-) denotes the ith row of the given matrix. In the special case 
m = 1, a vector b € B n .i if and only if all its entries are nonzero and have 
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different absolute values. Asking that B G B„ m is a fairly weak requirement: 
the complement of B„. m is an algebraic subset of R nxm , so B„, m is generic in 
the same sense as in Proposition 3.3.12. 

In this section, we will prove the following result: 

Theorem 8 The following two properties are equivalent: 

1. B G B„ iTO . 

2. The system (3.28) is strongly locally controllable around every state. 

If a system E is strongly locally controllable around x , in particular there 
exists some neighborhood W of x° so that every x E W is both controllable to 
x° and reachable from x°, that is, 

x° G int (n{x°)f]C{x )^ . (3.29) 

If E is strongly locally controllable around every state x Q , then the reachable 
sets lZ(x°) are both open and closed, for each a: . (Pick any x G TZ(x°); by (3.29) 
some neighborhood of x is included in TZ(x), and hence also in TZ(x°), so TZ(x ) 
is indeed open. Next take any x in the closure of TZ(x°); now we know that 
there is some neighborhood W of x such that every x G W can be steered to 
x. Since W n TZ(x°) ^ 0, we can find a point x which is reachable from x° and 
can be steered to x, so x G TZ(x°), proving that TZ(x°) is also closed.) Note that 
TZ(x°) is always nonempty, because x° G TZ(x°). Thus: 

Lemma 3.8.3 If a system E is strongly locally controllable around every state 
and X is connected, E is completely controllable. □ 

Putting together Theorem 8 and this Lemma, we have, then: 

Corollary 3.8.4 If B G B„. m , the system (3.28) is completely controllable. □ 

Theorem 8 will be proved after some generalities are established. 

A Local Controllability Lemma 

We start with an observation that is of independent interest. By co V we mean 
the convex hull of a set V . 

Lemma 3.8.5 Consider a continuous-time time-invariant system x = f(x,u) 
and suppose that x° G X is a state such that 

G intco {f(x°,u) \ ue 11} . (3.30) 

Then 



G int (n(x°)f]C( 



.r°- 
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Proof. Pick a finite subset Ho C II so that G intco {/(a; , u) \ u G Ho}. We 
assume, without loss of generality, that x° = 0. Let S 1 be the unit sphere 
{A G K™ | ||A|| = 1}. Define a function ip : 1 i-> E by letting 

<p(x) = max{h(A, cc) | A G S 1 } , 

where 

h(A, x) = min{(A, f(x, u)) \ u G llo} . 

The function h is a minimum of a finite collection of continuous functions, so h 
is continuous; thus, ip is well-defined and continuous as well. 

Since is an interior point of the convex hull of the set {/(0, u) \ u G Ho}, 
there is some S > such that, for every A G S , the vector — 4<5A is a convex 
combination of the /(0, u), u G Uo- Given any A G S , the number —AS is 
equal to (A,— 4<5A), which is a convex combination of the numbers (A,/(0,m)), 
u G Ho- So at least one of these numbers is < —AS. So h(A,0) < —AS. Since 
this is true for all A G S , we conclude that v?(0) < —AS. Since ip is continuous, 
there exists a > such that, if B = B a (x) = {x G K" | ||x|| < a}, then 8CI 
and ip(x) < —2(5 whenever j;G8. 

Now fix a point x G B, and let 5 be the set consisting of all those pairs 
(X, w) such that I C [0, 00 ) is an interval containing 0, and lu : T — > Ho is 
a measurable function (i.e., a control with values in Ho) for which there is a 
solution £ : I — > X (defined on the complete interval) of £(£) = /(£(£), w(£)) 
with initial state £(0) = 5, and this solution satisfies: 

d Jm < _, ( 3.3i) 

for almost every (£l. (Note that ||£(-)|| is absolutely continuous, because £ is, 
as is clear from the definition of absolute continuity.) 
We order S, by letting 

(Xl,Wl) ^ (X 2 ,U>2) 

iff Ii C I2 and wi is the restriction to T\ of u>2- Every totally ordered nonempty 
subset of S has an upper bound in S (take the union of the intervals, and a 
common extension of the a/s). By Zorn's Lemma, S has a maximal element 

In general, for any pair (J7", 1/) G S and its corresponding path Q, Equa- 
tion (3.31) implies that ||C(^)|| < IIC(0)ll — <^ f° r a H t € J, and, in particular, 
that ||C(*)|| < ||C(0)|| < a for all t £ J. So C is entirely contained in B. As 
* < f f° r all t £ J, J must be bounded. Furthermore, (closj',^) is also in 
S, when we view v also as a control defined on the closure of J , because the 
maximal solution for v seen as such a control cannot be £, but must extend 
to the closure (since £(•) takes values in a compact, cf. Proposition C.3.6), and 
Property (3.31) still holds a.e. on the closure. Applied to J = X, and since X 
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is maximal, this argument shows that X = closX. Thus, X = [0, T] for some T. 
Let £ be the path corresponding to (T,lo). We claim that 

£(T) = 0. (3.32) 

Indeed, suppose this claim would not be true, so x := £(X) G 23 is nonzero. 
Since </?(£) < ~~ 25, we have h(A,x) < —25, where A := — x/||x||. So there exists 
some uq G Uo such that (X, f(x,Uo)) < — 25. Let £ : [T, T + /3] — > X be the path, 
with £(T) = x, which corresponds to the control that is constantly equal to uq, 
for some (3 > 0. We may assume that £(<) ^ for all t (otherwise, we could 
extend by £ and already have a control that drives x to zero). Using that the 
gradient of x <— > II x II is jp-n-, we have 

p(t) := |||C(t)ll = (A(t),/(C(t),«o)), 



where 

X(t) 



at) 

UCtoll 



is a continuous function with A(T) = A. Since p(T) = (X, f(x,uo)) < —25, we 
can assume, by taking /? small enough, that p(t) < —6 for t G [T, T + /3]. Letting 
T = [0,T + /3], we extend w to a control w : I — > Uo, defining w(i) := uo for 
t € (7", T + /?]. The path with initial state x is £ : I — > X, whose restrictions 
to [0,T] and [T, T + /3] are £ and £ respectively. As (X, a)) G 5 and (X, w) -< 
(X, w) but (X, a>) 7^ (X, w), we have contradicted the maximality of (X, u>). This 
contradiction proves claim (3.32). 

In conclusion, we established that every ie6 can be controlled to 0. We 
may also apply this argument to the reversed system x = —f(x,u), whose paths 
are those of the original system run backward in time (cf. Lemma 2.6.8). The 
hypothesis applies to the reversed system, because if the convex hull of the set 
{/(0,ii) | u G Uo} contains zero in its interior, then the same is true for the 
convex hull of {— f(0,u) \ u G Uo}- Thus, we find some other ball B- with the 
property that every x G 23 _ can be reached from 0. Then W = B fl 23_ is so 
that every state in W is both controllable to and reachable from 0. I 

Corollary 3.8.6 Under the same assumptions as Lemma 3.8.5, the system is 
strongly locally controllable at x . 

Proof. Given x Q , and given any neighborhood V of x , we may consider the 
new continuous-time system with state space V, same input value set, and same 
equations, but restricted to V. The restricted system has the property that 
there is an open set W which contains x° and is so that every state in W is 
both controllable to and reachable from x . Thus any two states in W can be 
steered to each other, and by definition the paths stay in the restricted state 
space, namely, the desired neighborhood V. I 
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Remark 3.8.7 The argument used in the proof of Lemma 3.8.5 is analogous 
to the methods of proof used in Section 5.7, relying upon a "control-Lyapunov 
function" V(x) (the function V(x) = \\x\\ ) so that min u W(x) • f(x,u) < — c 
for all x near zero, for some c > 0. If this property is weakened to merely 
asking that, for all nonzero x near zero, min M \7V(x) ■ f(x, u) < 0, then only 
asymptotic controllability (cf. Chapter 5) will be guaranteed to hold, instead of 
controllability. □ 

A Result on Convex Hulls 

The relevant properties of 8 are summarized follows. 
Lemma 3.8.8 The function 8 = tanh satisfies: 

1. 8 is an odd function, i.e. 8(—s) = — 8(s) for all s£M; 

2- #oo = lim s ^ +oc 8(s) exists and is > 0; 

3. 8(s) < #oo for all set; 

4. for each a, b £ R, b > 1, 

hm — —— — = 0. (3.33) 

s^+oo 6»oo - 8{s) 

Proof. The first three properties are clear, with 8^ = 1, so we need to prove 
the limit property (3.33). Note that cr(x) = (1 + e~ x )~ l satisfies 

^- = <T(r) + e r - t a(-r) (3.34) 

for all r, t, and that 1 — tanh a; = 2er(— 2x) for all i£i Thus 

1 - tanh(q + 6s) 2o 2 (i-6) s /«, , oh n n 
= a(—2a — lbs) + e e v ' a(2a + lbs) — > U 

as desired. I 

For each vector a G 1" and matrix B G IR™ xm , we write 

S & m ■■= {8 n (a + Bu), ueR m ] . 

Lemma 3.8.9 If B e B nm and aGl", then e intcoS'a.B- 

Proof. Since B £ B„, m , there is some uq £ R m such that the numbers bi := 
row i (_B)u are all nonzero and have distinct absolute values. (Because the set 
of u's that satisfy at least one of the equations 

rowi(£>)w = 0, roWi(B)u + row j(B)u = , roWi(B)u — towj(B)u = 
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for some i ^ j is a finite union of hyperplanes in K m .) Fix one such uq. As 
Sa.,Bu ^ Sa__B, it is enough to show the result for Buq instead of B. So we 
assume from now on that B = {b\, . . . ,6 n )' has just one column, and all its 
entries bi are nonzero and have distinct absolute values. 

Assume by way of contradiction that ^ intcoS'a.B. Using a separating 
hyperplane^, we know that there is a nonzero row vector c = {c\, . . . ,c n ) such 
that cO n (a. + Bu) > for all 116R. Writing a = (01, . . . , &„)', this means that 

n 

Y^ Ci 0(a» + biu) > VmgK. (3.35) 

i=\ 

We now prove that such an inequality cannot hold, if the bi are nonzero and 
have distinct absolute values, unless all the d are equal to 0. Since 9 is odd, we 
may assume that each bi > 0, since each term Ci9(at + biu) for which &, < can 
be rewritten as (— Ci)B{— o, + (— bi)u). Thus, reordering if needed, we assume 
that < &i < . . . < b n . Finally, dropping all those terms in the sum for which 
Ci = 0, we may assume that c\ 7^ (using smaller n, if necessary). Taking the 
limit in (3.35) as u — » —00 we obtain YH=i c i{— ^00) > 0. So from (3.35) we 
obtain 



E 



Ci {Boo - 0(a>i + biu)) < VtieR. (3.36) 



=1 



Therefore 



x „oo - 6{a t + bill) 



E 

i=2 



If we prove that each term in the sum converges to zero as m -» +00 then it 
will follow that c\ < 0. But this fact follows from property (3.33) (applied with 
a = cii — bidi/bi, b = bi/bi, and noting that s = Oi + b\U — > 00 as u — » 00). 

If we take instead the limit in (3.35) as u — > +00, we find that ^i=i c i$oo > 0. 
We therefore also obtain from (3.35) that: 

n 

^2 Ci {Boo + B{ai + biu)) > VtieR. (3.38) 

i=\ 

Letting v = —u and cii = —a>i, and using that 6 is odd, 

EBoo — B(a,i + hv) „ 

i=2 ^00 - 6>(oi + biv) 

Taking the limit as V — * +00 and appealing again to property (3.33), we conclude 
that also ci > 0. Thus c\ = 0, contradicting the assumption made earlier. I 



t Several basic facts about separation of convex sets are reviewed in the chapter on linear 
time-optimal control, see page 431. 
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A Necessity Statement 

Lemma 3.8.10 If B $ B„. m then there exists a nonzero A € W 1 so that 

sign A 9 n (Ax + Bu) = sign A Ax 

for all (x,u) g 1™ x WL m (with the convention signO = 0). 

Proof. We denote, for each i, bi := roWj(£>) and dj := towj(j4). There are 
three cases to consider: (1) some bi = 0, (2) bi — bj = for some i ^ j, and (3) 
bi + bj = for some i ^ j. 

In the first case, we let A := e, (ith canonical basis vector). The conclusion 
follows from the equality 

e\9 n {Ax + Bu) = 6{a,ix) 

and the fact that sign(9(v) = signw. In the second case, we let A := e^ — ej. The 
expression 

(ej — ej) 8 n (Ax+Bu) = 8(aiX+biu) — 8(a,jX+bju) = 6(aiX+biu) — 6(ajX+biu) 

is nonnegative if and only if a,iX > a,jX (using that 9 is monotonic), that is, when 
X'Ax = ax — ajX > 0, so also in this case we have the conclusion. Finally, if 
bi + bj = 0, we pick A := ej + ej. Then 

(e-i + ej) 8 n (Ax+Bu) = 9(aiX+biu) + 8(ajX+bju) = 8(aiX+biu) — 8(—ajX+biu) 
(since 9 is odd), which is nonnegative precisely when (a, + a,j)x > 0. I 

Proof of Theorem 8 

From Corollary 3.8.6 and Lemma 3.8.9, since f(x°,u) = 9 n (Ax° + Bu), we have 
that B G B nm implies strong local controllability at each x Q . To show the 
converse, suppose that B £ B„ iTO . Let A be as in Lemma 3.8.10 and let V be R" 
if \'A = and {x | X'Ax > 0} otherwise. Pick any state x° e V. Let TZ v (x°) be 
the set of states reachable from x without leaving V. Then 1Zv(x°) is included 
in the half space 

{x | X'x > A'x } . 

This is because, if £ = 9 n (A^ + Blu) and £(£) € V for all t, then 

^A'C(i) = A'4(A£(i) + Su;(i)) > 0, 

so the function <p(£) := A'£(£) is nondecreasing. Thus there cannot be a neigh- 
borhood of x° which is reachable from x° without leaving V. This completes 
the proof of the Theorem. I 
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Exercise 3.8.11 One could also define a class of systems as in (3.28) with other 
choices of 6. Theorem 8 may not be true for such other choices. For instance, 
the theorem fails for 6 = identity (why?). It also fails for 6 = arctan: Show that 
the 4-dimensional, single- input system 

X\ = arctan(xi + x-i + x% + 24 + 2u) 

±2 = arctan(xi + 22 + 2^3 + 24 + 12u) 

±3 = arctan(— 3m) 

±4 = arctan(— Au) 

satisfies that B € B„. m but is not controllable. Explain exactly where the 
argument given for 9 = tanh breaks down. □ 

3.9 Piecewise Constant Controls 

Often it is of interest to know how much more restrictive it is to consider only 
controls that are of a particular type, such as polynomial in time or piecewise 
constant. In general, for any m,cr,T, let A be any subspace of £™(a, r) that 
satisfies: 

1. For each u> G C^ there is an equibounded sequence in A converging to u>; 
and 

2. it contains all constant controls. 

For instance, one may take polynomial or piecewise constant controls (see Re- 
mark C.l. 2 in Appendix C). For linear systems, controls in A are rich enough: 

Proposition 3.9.1 Assume that £ is a linear continuous-time system, a < t, 
and A is as above. Then the following conditions are equivalent: 

(a) £ is controllable on [er, r]. 

(b) For each x, z G X, there exists some control in A such that </>(r, a, x, lu) = z. 

In particular, for time-invariant systems, controllability is equivalent to control- 
lability using controls in A. 

Proof. Fix any x G X. By Theorem 1 (p. 57), Part 2(i), the set 

{4>{t, a, x, w) I w e A} 

is dense in X. But this set is an affine subspace, since this is the set of all 
expressions of the form 

<f>(r, a,x,0) + (J){t, a, 0, ui) , 

and the map </>(r, a, 0, ■) is linear. Thus, it must be all of X. I 
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Proposition 3.9.2 Let E be a time-invariant continuous-time system of class 
C 1 with IX = R m , and assume that there is some equilibrium pair (x°,u°) such 
that the linearization of £ at (x°, u°) is controllable. Pick any a < r and any A 
as above. Then there exists a neighborhood V of x° such that for each x, z G V 
there exists some control uj € A such that </>(r, c, cc, w) = z. 

Proof. This is proved in exactly the same form as the local controllability 
Theorem 7 (p. 126), by application of the Implicit Mapping Theorem to the 
mapping a, with controls given the sup norm. The only modification is that 
one must restrict controls to A, seen as a (not necessarily dense) subset of 
£^(<7, t). But full rank is already achieved on these, because of Proposition 
3.9.1 and the fact that the differential is nothing other than the corresponding 
reachability map for the linearization. I 

Note that the above argument used the fact that a is C 1 with respect to 
the sup norm and that the Implicit Function Theorem was applied about the 
control constantly equal to u° (which is in A by hypothesis). 

Corollary 3.9.3 If E is as in Proposition 3.9.2, It = K m , and E is controllable, 
then for each x, z G X there exists some a < t and some piecewise constant 
control uj € w- a ' T > such that </>(t, <t, x,uj) = z. 

Proof. Pick first any a < r and apply the Proposition with A = family of 
all piecewise constant controls, to obtain the open set V. Let x, z be as in 
the statement. Because of the controllability assumption, there is some control 
mapping x into x°. By Lemma 2.8.2 there exists some piecewise constant control 
lui mapping x to some element x' in V , and similarly there exists a piecewise 
constant control L02 mapping some z' from V into z. Concatenating uj\ with 
a piecewise constant control sending x' to z' and this in turn with u>2, the 
conclusion follows. I 



3.10 Notes and Comments 

Controllability of Time-Invariant Systems 

Since the early work on state-space approaches to control systems analysis, it 
was recognized that certain nondegeneracy assumptions were useful, in particu- 
lar in the context of optimality results. However, it was not until R.E. Kalman's 
work (see, e.g., [215], [216], [218], and [231]) that the property of controllabil- 
ity was isolated as of interest in and of itself, as it characterizes the degrees of 
freedom available when attempting to control a system. 

The study of controllability for linear systems has spanned a great number 
of research directions, and topics such as testing degrees of controllability, and 
their numerical analysis aspects, are still the subject of intensive research. 

The idea in the proof of Corollary 3.2.7 was to use an ascending chain con- 
dition on the spaces Xi to conclude a finite-time reachability result. Similar 
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arguments are used often in control theory, under other hypotheses than finite 
dimensionality (or finiteness, as in Lemma 3.2.4). For instance, for discrete- 
time linear systems over Noetherian rings, one may use the same idea in order 
to conclude that IZ(x) = lZ T (x) for large enough T. 

Controllability of continuous-time piecewise linear systems is studied in [267] 
and [411]. 

There is also an extensive literature on the controllability properties of in- 
finite dimensional continuous-time or discrete-time linear systems; see, for in- 
stance, [107], [147], and [291]. For such systems it is natural to characterize 
"almost" reachability, where the set of states reachable from a given state is 
required only to be dense in the state space. 

Algebraic Facts About Controllability 

What we called the "Hautus condition" actually appeared first in [321], Theorem 
1 on page 320, and in a number of other references, including [40]. However, [175] 
was the first to stress its wide applicability in proving results for linear systems, 
as well as in extending the criterion to stabilizability (asymptotic controllability) 
in [176]. Sometimes the condition is also referred to as the "PBH condition" 
because of the Belevitch and Popov contributions. 

It should be emphasized that genericity of controllability was established 
only with respect to the set of all possible pairs. Often one deals with restricted 
classes of systems, and among these, controllable systems may or may not form 
an open dense subset. This gives rise to the study of structural controllability] 
see, for example, [298], and more generally the book [350] and references therein, 
for related questions. 

In structural controllability studies, classes of systems are defined typically 
by constraints on the matrices (A,B). For example, any system obtained from 
an equation of the type 

x + ax + f3x = "fu 

(as arises with a damped spring-mass system) via the introduction of state 
variables X\ := x,X2 := x will have An = B\\ = and A\2 = 1. These 
coefficients 0, 1 are independent of experimental data. Graph-theoretic methods 
are typically used in structural controllability. 

Controllability Under Sampling 

Necessary and sufficient conditions for the preservation of controllability under 
sampling for multivariable (m > 1) systems are known, but are more compli- 
cated to state than the sufficient condition in Theorem 4 (p. 102). See, for 
instance, [154]. 

The dual version of the Theorem, for observability (see Chapter 6) , is very 
closely related to Shannon's Sampling Theorem in digital signal processing. 

Some generalizations to certain types of nonlinear systems are also known 
(see, for instance, [208] and [371]). 
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More on Controllability of Linear Systems 

The result in Exercise 3.5.7 is a particular case of the more general fact, for 
time-invariant continuous-time single- input (m = 1) systems, that the operator 
that gives the optimal transfer from to a state x in time s has an operator 
norm 

||L#|| =0(e _n+ 5) 

for small e. This is proved in [345], where the multiple-input case is also char- 
acterized. 

Bounded Controls 

Theorem 6 (p. 117) can be found in [343]. A particular case had earlier been 
covered in [266], and the general case, as well as the obvious discrete-time ana- 
logue, were proved in an abstract algebraic manner in [364]. We based our 
presentation on [170], which in turn credits the result, in the stronger form 
given in Corollary 3.6.7, to the earlier thesis [196]. 

It is also possible to give characterizations of controllability under other 
constraints on control values. For instance, positive controls are treated in [60]. 

First-Order Local Controllability 

The problem of characterizing local controllability when the first-order test given 
in Theorem 7 fails is extremely hard, and constitutes one of the most challenging 
areas of nonlinear control research. It is often of interest to strengthen the 
definition of local controllability: For instance, about an equilibrium state x 
one might require that any state close to x be reachable from x in small time 
(small-time local controllability). Such stronger notions give rise to interesting 
variations of the nonlinear theory; see, e.g., [388] and references there for this 
and related issues. 

Recurrent Nets 

The systems studied in Section 3.8 are often called "continuous-time recurrent 
neural networks" . The motivation for the term comes from an interpretation of 
the vector equations for x in (3.28) as representing the evolution of an ensemble 
of n "neurons," where each coordinate X{ of x is a real-valued variable which 
represents the internal state of the ith neuron, and each coordinate Ui,i = 
1, . . . , m of u is an external input signal. The coefficients A^j, Bij denote the 
weights, intensities, or "synaptic strengths," of the various connections. The 
choice 8 = tanh, besides providing a real-analytic and globally Lipschitz right- 
hand side in (3.28), has major advantages in numerical computations, due to 
the fact that its derivative can be evaluated from the function itself (6 1 = 1 — 
2 ). (Sometimes, however, other "activation functions" are used. One common 
choice is the function a(x) = (1 + e~ x )~ 1 , which amounts to a rescaling of 9 to 
the range (0, 1).) Among the variants of the basic model (3.28) that have been 
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studied in the literature are systems of the general form x = —x + 9 n (Ax + Bu), 
where the term —x provides stability. Recurrent nets (or these variants) arise 
in digital signal processing, control, design of associative memories ("Hopfield 
nets"), language inference, and sequence extrapolation for time series prediction, 
and can be shown to approximate a large class of nonlinear systems; see e.g. 
[237]. 

The implication 2 => 1 in Theorem 8 is from [375] , where the reader may also 
find a solution to Exercise 3.8.11. The necessity of the condition B G B„ !m was 
shown to the author by Y. Qiao. A characterization of a weak controllability 
property for the discrete-time analogue of these systems had been given earlier, 
in [12], which had also obtained partial results for the continuous-time problem. 

Piecewise Constant Controls 

Results similar to Proposition 3.9.1 are in fact true for much larger classes 
of systems. It is possible to establish, for instance, that piecewise constant 
controls are enough for any analytic continuous-time system. In [367], it is 
shown that under one weak extra assumption, polynomial controls can be used 
for controllable analytic systems. See [163] for a survey of such results. This 
section gave only very simple versions of these more general facts. 



Chapter 4 

Nonlinear Controllability 



In this chapter* we study controllability questions for time-invariant continuous- 
time systems x = f{x,u). 

The property that is easiest to characterize completely for nonlinear con- 
tinuous-time systems is the accessibility property sometimes also referred to as 
weak controllability. This is the property that one should be able to reach from 
any given state a set of full dimension. However, for certain restricted classes 
of systems, results on complete controllability are also available. 

To begin, we establish some basic facts regarding Lie algebras of vector fields. 
(Working on open subsets of Euclidean spaces, we do not make any explicit use 
of notions of differential geometry. ) 



4.1 Lie Brackets 

If / = (/l) ■ • ■ ! In)' '■ C — * H& n is a continuously differentiable map defined on 
some open subset O C W, /* denotes the Jacobian of /, thought of as a matrix 
function on O. That is, for each x° e O, f*{x°) s R™ xp is the Jacobian of / 
evaluated at x° , the matrix whose (i, j)-th entry is -^ L \ x — x o. 

In the special case when p = n, one calls a continuously differentiable map 
/ : O — > R™ a vector field defined on the open subset O of R™. A smooth vector 
field is a smooth (infinitely differentiable) / : O — » R"; from now on, and unless 
otherwise stated, when we say vector field we will mean "smooth vector field" . 
The set of all (smooth) vector fields on a given O C R™ is denoted by V(O). 
It is a real vector space under pointwise operations, that is, (rf + g){x) := 
rf{x)+g{x) for all r eR, f,g e V(£>), and xeO. 

When ip : O — » R is differentiable, (p*(x) = Vy(x) is a row vector, the 
gradient of ip evaluated at x. The set of smooth functions O — > R is denoted by 
F(0). This is also a real vector space under pointwise operations. 



*The rest of the book, except for Section 5.3, is independent of the material in this chapter. 
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For each / £ V(£>) and each <p £ ¥(0), Lfip £ ¥(0) is the directional or Lie 
derivative of ip along /: 

{L f ip){x) := <p*(x)f(x). 

One may view Lf as a linear operator F(O) — ► F(O). Two vector fields f,g £ 
V(O) are equal if and only if Lf = L g . (Because, if Lfip = L g ip holds for all <p, 
then it holds in particular for each of the n coordinate functions 7Tj(x) = CEj, and 
hence the ith coordinates of / and g coincide: fi(x) = (LfKj)(x) = (L g iVi)(x) = 
gi(x).) Note that Lf is a first-order differential operator, while the composition 
Lf o L g , which we write simply as LfL g , is a second-order operator: one can 
verify easily that 

LfL g ip = g'H v f + (p*g*f , (4.1) 

where H v is the Hessian matrix I dx .Q x . ) • Since H v is symmetric, 

LfLgP - LgLfP = Lg^f^f^gip. (4.2) 

Definition 4.1.1 The Lie bracket of f,g £ V(O) is [f,g] := g*f - f*g £ 
V(O). ' □ 

Equation (4.2) says that 

L U,a\ = L f L a ~ L 9 L f ( 4 - 3 ) 

for all vector fields / and g. The binary operation /,j h [/, g] is skew- 
symmetric: [f,g] = -[g,f] and bilinear: [r/i + / 2 ,#] = r[/i,g] + [f 2 ,g} and 
[/j^i + 52] = r[f,gi] + [f,g 2 ]- It is convenient to write 

ad /5 := [/,<?] (4.4) 

and to think of ad/, for each fixed / £ V(O), as a linear operator V(O) — » V(C). 
This operator is a differentiation operator with respect to the Lie bracket: 

ad /[5>^] = [ ad /9> /»] + [5: ad /fr] 

for all f,g,h, a formula also known as the Jacobi identity, especially when 
written in the equivalent form 

[f,[g,h\] + [h,[f,g]] + [g,[h,f]} = 0. 

This is immediate from the equality 

L [f,[gM\ = L f L a L h ~ L f L hL g - L g L h L f + L h L g L f 

obtained by using Equation (4.3) twice and using similar expansions for the 
other two terms. 

Some properties of, and relations between, the operators just introduced are 
as follows. 
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Lemma/Exercise 4.1.2 For any f,g £ V(O) and any cp,ip £ F(0), 

• L^fip = tpLftp, 

• [<pf,ip9] = <ftp[f,g] + {Lfip)pg - (L g <p)ipf. 

Definition 4.1.3 A Lie algebra (of vector fields on O) is a linear subspace 
S C V(O) i/iai is closed under the Lie bracket operation, that is, [f,g] £ S 
whenever f and g are in S. □ 

For any subset A C V(0), we define Ala, the Lie algebra generated by A, 
as the intersection of all the Lie algebras of vector fields which contain A. (The 
set of all such algebras is nonempty, since it includes Y(0).) An intersection of 
any family of Lie algebras is also a Lie algebra; thus, 

-^LA = smallest Lie algebra of vector fields which contains A . 

Lemma 4.1.4 Let A be a subset of V(O). Denote Aq := A, and, recursively, 

A+i := {[f,g] \f£A k ,g£A} , fc = 0,l,2,... , 

as well as Aoo '■= Ufe>o ^ k - Then, *4la is equal to the linear span of A^,. 

Proof. Any Lie algebra which contains A must contain „4oo (because, induc- 
tively, it contains each Ak), and hence also contains its linear span, which we 
will denote by A. So we must only show that A is a Lie algebra, i.e., that it is 
closed under the Lie bracket operation. Since [X, ■] is linear, it suffices to show 
that 

X£AandY£A k => [X,Y]£A (P k ) 

holds for every k. Observe that, since each X £ A can be written as a linear 
combination of elements of .Aoo, an d since [-,Y] is linear, property (Pk) is equiv- 
alent to the statement that [X, Y] £ A whenever X £ A^ and Y £ Ak- We 
show (Pk) by induction on k. 

The case k = is clear since, for any / £ Aq = A, by definition [X, f] £ Ai+i 
if X £ Ai- Assume now that the result has been proved for all indices less than 
or equal to k, and pick any X £ A and any Y £ Ak+i- Thus we may write 
Y = [Yo, /], for some f £ A and Yq £ Ak- The Jacobi identity gives: 

[X,Y] = [X,[Y ,f]} = [[X,Y ],f]-[[X,f],Y ]. 

By inductive assumption, [A, Yo] £ A, so (induction again) [[A, lb]i/] € A. 
Similarly, by induction we know that [[A, /],Yo] £ A. As A is a subspace, 
[A, Y] £ A, as desired. I 
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The Lemma says that every element in the Lie algebra generated by the set 
A can be expressed as a linear combination of iterated brackets of the form 

[[• • • [a j yy , /3] , ■•■)/*], 

for some fi G A. (We make the convention that an "iterated bracket of 
length £ = 1" is just an element /i of the generating set A.) Equivalently, 
by skew symmetry, one can write any such element as a combination of brackets 
[fli ■ • ■ j [/3) [/2j fi]]]i that is, as adf e . . . adf 2 fi. So Ala is the smallest linear 
subspace of V(O) which includes A and is invariant under the linear operators 
ad/, f e A. 

A Digression on Linear Algebra 

Before proceeding, we make a simple but useful observation in linear algebra. 
We fix any k = 0,1, ... ,00 or k = lu (as usual, we let C w denote the set of 
analytic functions). 

Lemma 4.1.5 Suppose that Q : W —> W ixp is a matrix function of class 
C k , defined on an open subset W C R m , which satisfies that, for some q, 
rankQ(w) = q for all w G W. Then, for each wq G W, there is some neigh- 
borhood Wq of Wo, and there are two matrix functions C : Wq — > K" xn and 
D : Wo —> K pxp , both of class C k and nonsingular for each w G Wo, such that 

C(w) Q{w) D{w) = I 1 9 ) for all w e W Q , (4.5) 







where I is the q x q identity matrix. 



Proof. Pick any wq G W. Since Q(u>o) has rank q, there exist two permutation 
matrices Pi G R nxn and P^ G W }Xp so that the submatrix obtained from the 
first q rows and columns of PiQ(u>o)P2 is nonsingular. 

Now let A(u>) be the submatrix obtained by selecting the first q rows and 
columns of P\Q(w)P2, for each w G W, that is, 

««->* - {V : 

(where asterisks indicate arbitrary functions of w of class C k ). Thus A(w) 
is a matrix function of class C k , and is nonsingular at w = Wo- Since the 
determinant of A(w) is a continuous function of w, there is some open subset 
Wo of W such that det A(w) ^ for all w G Wo- On Wo, we consider the 
matrix function 

n r \ i>n( \p ( ^M" 1 0\ / / X( w )\ 
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where I is an identity matrix of size q x q and X(w) and Y(w) are of class C k . 
Note that rank<3i(u>) = rankQ(w) = q for all w. So this matrix has rank = q: 



I 
-Y(w) 



Qi(w) 



-X(w) 
I 



I 
* 



and therefore the "*" block is identically zero. In conclusion, 







Pi 



Q(w) 



Y{w) I / 
and the proof is complete. 



Pi 



A(w)- 1 




-X{w) 
I 



I 




Corollary 4.1.6 Suppose that Q : W — > K" xp and / : W — > W 1 are a matrix 
and a vector function, both of class C k , defined on the same open subset of 
some space M m , and that Q(w) has constant rank on W. Then, the following 
two properties are equivalent: 

• For each w G W, f(w) belongs to the subspace of K n spanned by the 
columns of Q{w). 

• For each wq G W, there exist a neighborhood Wo of wq and a class C k 
function a : Wq — > W, so that 



f(w) = Q(w)a(w) for all w £ W . 



(4.6) 



Proof. The second property obviously implies the first, so we need only to prove 
the other implication. Thus, assume that rank (Q(w), f(w)) = rankQ(w) = q 
for all w G W, and pick any wq € W. By Lemma 4.1.5, there exist a neighbor- 
hood Wo of wq, and C(w) and D(w) of class C k on Wo, so that Equation (4.5) 
holds. Note that the matrix (C(w)Q(w)D(w),C(w)f(w)) has the same rank as 
the matrix (Q(w), f(w)) (since C(w) and D(w) are nonsingular) , so 



rank 



I 




C(w)f(w] 



for all w. We conclude that the last n — q coordinates of C(w)f(w) must vanish 
that is, there is a class C k function ao : Wo — * K 9 so that C(w)f(w) = ( 



on Wq, where 0„_ g 



l«sothatC(w)/(w) 
is the zero vector in M" -9 . We conclude that 



C(w)Q(w)D{w) 



a (w) 



Jp-q 



C(w)f(w), 



and therefore, since C(w) is nonsingular, that a(w) := D{w) 
desired. 



a (w) 
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Tangent Vectors 

Let W C W and O C K" be open subsets of Euclidean spaces. We will call a 
smooth map 

M : W -» O 

a sZice if its Jacobian Af*(to) has rank p at each w G W. The vector field 
/ G V(O) is said to be tangent to M if, for all w G W, f(M(w)) is in the 
subspace of R n spanned by the columns of M*(w). 

Remark 4.1.7 Geometrically, we think of the image of M as a "p-dimensional 
smooth slice" of O. In differential-geometric terms, when M is one-to-one we 
are providing a chart for the submanifold represented by this image. Tangent 
vector fields in the sense just defined are precisely those which, when restricted 
to the submanifold, are tangent to it in the usual sense. □ 

By Corollary 4.1.6, applied to the composition foM:W—> R", / is tangent 
to M if and only if for each wq G W there is some neighborhood Wo of wq and 
a smooth function a : Wo — > W so that 

f{M{w)) = M^{w)a{w) for all w G W . (4.7) 

It is clear that the set of all vector fields tangent to M forms a linear subspace 
of V(O). One of the most useful facts about Lie brackets is expressed in the 
next result. 

Lemma 4.1.8 Let M : W —> O be a slice. Then the set of all vector fields 
tangent to M is a Lie algebra. 

Proof. Pick any two tangent vector fields /, g and any wo G W. We must 
show that [f,g](M(wo)) belongs to the subspace of K™ spanned by the columns 
of M*(wo). Pick a (common) neighborhood Wq of wo and smooth functions 
a : W -» R p and /3 : W -* K p so that f(M(w)) = M.,{w)a{w) and g{M{w)) = 
M*{w)(3(w) for all w G Wq. We will prove the following formula, from which 
the conclusion will follow: 

[f,g](M(w)) = M*(w)[a,P](w) for all w G Wo . (4.8) 

(Note that [/, g] is a Lie bracket of vector fields in K", and [a, j3] is a Lie bracket 
of vector fields in W. The equation amounts to the statement that vector fields 
transform covariantly under smooth maps.) From now on, when we write w we 
mean "for each w G Wq" . Consider: 

f(w) :=f(M(w)) = M*(w)a(w), g{w) := g(M(w)) = M*{w)f3{w) . (4.9) 

The chain rule gives 

(/)» = U{M{w))M*{w) , (£)» = g*{M{w))M*{w) . (4.10) 
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We pick any i G {1, . . . , n}, let e; be the ith canonical basis vector, and consider 
the smooth function 

<Pi := e-M : W -> R . 
From (4.9) we have that 

e'j = L a tpi and e' i g = L ip i . (4.11) 

So 

ejfa./) (MH) = e !j,(MW)M,(»)aH 

= e^ (</)„,(«;) a(w) = (e^fif) i|c (w) a(w) using (4.10) 

= {Lpipi)^) a{w) = {LuLpipi) (w) using (4.11). 

An analogous argument gives that 

e' l (f,g)(M(w)) = {L p L aVi ){w) 

so we conclude, from the fact that £[«,/?] = L a Lp — LpL a (Equation (4.3), 
applied to a and (3) that 

e^[f,g](M(w)) = 4M*(w)[a,0\(w). 

As i was arbitrary, (4.8) holds. I 

4.2 Lie Algebras and Flows 

We turn next to establishing a connection between Lie algebras of vector fields 
and sets of points reachable by following flows of vector fields. 

Some Additional Facts Concerning Differential Equations 

For an open subset O C K™ and a (continuously differentiable but not necessarily 
smooth) vector field f : O —> R™, we write 0, or (j)f if we wish to emphasize /, 
to denote the flow of /. This is the map that associates to (t,x°) 6RxO the 
solution <p(t, x°) = x(t) at time t € R, if defined, of the initial value problem 

x = f( x ), x(0) =x° . 

(If the solution of this initial value problem does not exist on an interval con- 
taining and t, 4>{t,x°) is undefined.) We denote by T>f, or just P if / is clear 
from the context, the domain of </>. This is a subset of R x O which contains 
{0} x O. 



(t,x)e 

and 

(t,x)€ 
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Exercise 4.2.1 Let O = M. and f(x) = x(l — x). Show that T> is the union of 

Ix [0,1], 

x < and t < In 



, x ~ ! 
X > 1 and £ > In 

Graph D as a subset of K 2 , and find an explicit formula for the flow (f>(t,x). □ 

The set T> is always open and the map <ft : T> — > O is continuous. This is 
a corollary of Theorem 1 (p. 57), arguing as follows. (See also Remark 3.7.3.) 
Let (t, x ) G T>, and pick any numbers a < < b so that [a, 6] x {x a } C D and 
£ G (a, 6). The map 

O -» C ([o,6],O) : z h-> restriction of <^(-,z) to [a, 6] (4.12) 

is defined for all z in some neighborhood Co of x° , and it is continuous when 
C°([a, 6],C) is endowed with the uniform convergence norm. This is shown in 
Theorem 1. (The proof there applies only to the interval [0,6], but the result 
can also be applied to the "reversed-time" system x = —f(x), and this gives 
the conclusion for the interval [a, 0] as well; thus, the result is also true on 
[a, b].) In particular, it follows that (a, b) x Oq C T>, so the domain T> is open. 
To prove continuity, assume that (tk,Zk) — > (t,X ) as fc — > oo and take any 
£ > 0. Since (4.12) is continuous, there is some K such that k > K implies 
</>(s,cc°) — 4>(s,Zk)\ < e/2 for all s € [a, b], and since <p(-, x°) is continuous, we 
may also assume that \(j)(tk,x ) — (j>(t,x°)\ < e/2 for all such k. Furthermore, 
we may assume that tk € [a, b]. Thus 

|0(t fe ,z fc )-0(£,a; o )| < \<f>(t k ,z k ) - <j>{t k ,x°)\ + \4>{t k ,x a ) - <f>(t,x°)\ < e 

for all k > K. 

Since / is of class C , the map <f> : T> — » O is in fact continuously differen- 
tiable. Like continuity, this is also a corollary of Theorem 1 (p. 57), where we 
now argue as follows. It is shown in that Theorem (as before, the result was only 
proved for positive times, but the generalization to r < is immediate) that, 
for each t, the map z t— > </>(t, z) is continuously differentiable, and its partial 
derivative with respect to the ith coordinate, evaluated at z = z°, is 

^W) = A z .(r), (4.13) 

where A z o(-) is the solution of the variational equation A(£) = f*(4>{t, z°))X(t) on 
[0, t] (or [r, 0], if r is negative) with initial condition A(0) = e i; the ith canonical 
basis vector in 1™. 

Moreover, q~{t,Z ) = A z o(r) depends continuously on (r, z°). Indeed, 
suppose that (r k ,z k ) — > (r, z°). As a first step, we remark that A Zfc (s) con- 
verges uniformly to X z o(s), for s g [a, 6]. This can be proved by noticing 
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that the same proof that was used to show that X Zk (t) converges to X z o (t) 
for each t, that is to say the continuity of z i— > g^:(t,z), gives automatically 

that z i— » g^(-,z) is also continuous as a map into C°([a, 6],R n ); see the argu- 
ments after Equation (2.40). (Another proof is by viewing A = f*(4>(t, x))X as 
a system A = /„(ii)A where u = 4>(t,x) is a control, and using continuity with 
respect to controls.) As a second step, we point out that X z o(Th) — > A 2 o(t), 
because A z <> is continuous, being the solution of a differential equation. Writing 
A^ fc (r fe ) - A z o(r) = A^ fc (r fe ) - A z o(rfe) + A z o(T fe ) - A z o(r), we conclude continuity 
as earlier. 

To show that <j> is C 1 , we need to prove also that -W| is continuous. But this 
is clear from the facts that -£(t,x) = f((p(t,x)) and <f>(t,x) is continuous on 
(t,x). 

Observe also that, being the solution of a differential equation, A z o(-) is 

differentiable, from which it follows that dtd , exists. Similarly, -^ exists, 
because §if((f>(t,x)) = /*(</>) f£. 

Lemma 4.2.2 Pick any k = 1, 2, . . . , oo. If / is of class C , then <f> is of class 
C fc . 

Proof. The case k = 1 was established in the previous discussion. By induction, 
assume that the result has been proved for all 1 < £ < k < oo; we now prove it 
for k as well. Consider the following differential equation in O x W 1 : 



d_ ( x \ ( f(x) \ (x 

dt \X \f*(x)Xj * \ X 



(4.14) 



Observe that F is of class C k 1 and k — 1 > 1. For any fixed i = 1, . . . ,n, 
letting ei be the fth canonical basis vector, 

4> F {t,{x,ei)) = ( d£,' X L 

The inductive hypothesis applies to Equation (4.14), implying that the flow 
4>f{-, (■, •)) i s of class C k ~ 1 , so also 4>f{-, {■, e,)) is C k ~ l , from which we conclude 
that -j^r is fc— 1-times continuously differentiable. Thus every partial derivative 

of order k which starts with -Jr— exists and is continuous. 

axi 

If we show that -£ is also C fe ~ 1 , it will have been proved that cp ls C • To 
see this, consider the new extended system, also in O x W 1 , 

s(:) -U&)) - o (;)- ( " 5 ' 

Since x = f(x) implies x = f*{x)f(x), we have that 
4> G (t,(x,f(x)j) = (^j) 
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and, by induction, <pc is of class C k ~ x , so -£ is indeed C k ~ 1 . I 

Remark 4.2.3 It is tempting to try to show the continuity of -^- (case k = 1) 
by the same argument, using Equation (4.14). However, a problem with that 
argument is that F would only be assured of being continuous, so an additional 
assumption is required (for instance, that / is C 2 ). The argument given for 
k = 1, before the Lemma, which views x as an input to the second equation, 
avoids this assumption. Yet another argument would be based on the general 
fact that solutions of ode's depend continuously on initial data provided only 
that the right hand side be continuous - not necessarily Lipschitz - and that 
the equation be known to have unique solutions (as Equation (4.14) does, even 
if / is only C , because of its special triangular structure). □ 

Exercise 4.2.4 Show that, if / is of class C , all the partial derivatives of <j) 
of order k are differentiable with respect to t. [Hint: Show, by induction, that 
every such derivative is also the solution of a differential equation.) □ 

Exercise 4.2.5 Find explicitly the extended flow 4>p associated to the system 
in Exercise A. 2.1. □ 

Lie Brackets and Accessibility: Motivational Discussion 

We prefer to use, in this discussion, the alternative notation 



e tf x° 



4>(t,x°), (4.16) 



which is more convenient for dealing with compositions of flows corresponding 
to different vector fields. For instance, we can write e t2 ^e tl9 x instead of the 
more cumbersome (j)f(t2,<f> g {ti,x)). (Note that, when the differential equation 
happens to be linear, i.e. x = Ax and A is a matrix, e'^x is precisely the same 
as ex, where e is the exponential of the matrix A; in general, however, e f fx 
is merely a useful notational convention.) 

The idea behind the use of Lie algebraic techniques is as follows. Suppose 
that our goal is to describe the set of points z° that can be reached, from a 
given starting point x° , by following the flow of two vector fields / and g, which 
may represent the motions corresponding to two different constant controls U\ 
and U2, for different intervals of time. For instance, one may follow / forward 
for 7 units of time, then g forward for 2 units, and finally / backward for 3 
units. (That is, we go from x° to e _3 ^e 2s e 7 ^x°.) The question of reachability is 
nontrivial, because the motions along / and g need not commute. (The resulting 
state is not the same, in general, as e 29 e 4 'x°.) 

An intuitive and geometric example of this phenomenon is as follows: let / 
induce a counterclockwise rotation (at 1 rad/sec) about the z-axis in M 3 and g 
a counterclockwise rotation about the y axis (also at 1 rad/sec), that is, 



f(x) = xi and g(x) 
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The flows are 



and 



e tf x 



e tg x 




Then, the sequence of rotations: / for n/2 sec, g for ir/2 sec, and finally / for 
— 7r/2 sec (i.e., rotate clockwise) has a different net effect than that of simply 
rotating about the y axis for tt/2 rad. It produces instead a clockwise 7r/2-rad 
rotation around the x-axis, as is easy to see geometrically, or algebraically, since 

e (-7r/2)/ e ( 1 r/2)g e (7r/2)/ a . equals 




1 





°\ 


lx\ 








1 


X2 





-1 


0/ 


\%3 



This example shows that a totally new motion, different from a pure rotation 
around the y or the z axes, results by combining the original flows. 

Lie theory provides a way to approach the question, by asking the following 
"infinitesimal" version of the problem: In what effective directions can one 
move, from the given initial point x°? Clearly, one can move in the direction of 
f(x), by following the flow of / (more precisely, one can move along a curve 
j(t) = 4>(t, x ) whose tangent 7(0) at the initial time is f(x )), and similarly 
one can move in the direction of g(x ). On further thought, it is easy to see 
that one may also move in the direction of f(x°) + g(x°), in the following sense: 
there is a curve 7 with the property that 7(f) is in the reachable set from x° , for 
all t small enough, and such that 7(0) = x Q and 7(0) = f{x°) + g(x ). Indeed, 
it is enough to take the curve 7(f) := e tg e t ^x° , calculate 

7(t) = ^(*.M*.* )) + t(^/(M«))|(M°) 

and use that -4^-(0,a; ) = /. More generally, one may find curves which allow 
movement in the direction of any linear combination of f(x°) and g(x°). The 
interesting observation is that other directions, in addition to those lying in the 
linear span of {f(x°),g(x )}, may appear, namely, directions obtained by taking 
Lie brackets. For example, in the case of the rotations discussed above, 



[/,0]O*O 
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which is the vector field whose flow is the one that corresponds to the clockwise 
rotation about the x axis which we had discovered before. The result in the 
following exercise shows the role of Lie brackets in general; we do not need the 
result in this form, though it will be implicit in several arguments. 

Exercise 4.2.6 Show that, for any two vector fields / and g, and any state x°, 
the curve 

7 (t) : = e'^e-^e^e^x (4.17) 

(defined for all t in some neighborhood of t = 0) has the property that 7(0) = 
[/, g](x°). (Hint: You may want to use the expansion, for x(t) = e*^a; , 

X (t) = x(0) + tx(0)+^x(0) + o(t 2 ) = x° + tf(x°) + t -f*(x°)f(x°) + o(t 2 ) 

(and similarly for g), as well as Taylor expansions to first order for each of f(x) 
and g(x), to show that e -^9 e -Vif e Vig e Vif x o = e t[f,g] x + ^ as £_►().) □ 

Nonsingular Reachability 

Assume given an open set O C R" and a set of vector fields A C V(O). 

Definition 4.2.7 A k-tuple (fi, ■ ■ ■ , fk) of elements of A is nonsingular at 
x° e O if there exists some vector to = (£?, • • • ,t1) € M> such that the map 

Ffl_ fk : Df^ Jk ^O : t = (h, . . . , t k ) ~ e'*'* . . . e^e^x , 

which is defined on some open subset D x f * ofM. k containing t = (0, ... ,0), 
has Jacobian of rank k at (t®, . . . , t®) . □ 

Notice that since, by Lemma 4.2.2, e f f x° is C°° on (t,x°), one knows, in- 
ductively, that each map FT * is C°°. 

Recall that Ala is the Lie algebra generated by the set A. For each x° £ O, 
we consider the subspace 



A LA (x°) := {X(x°),X £ A LA } 



of R™. The key result is the next one. 

Lemma 4.2.8 If A-la( x °) = K™ then there is some nonsingular n-tuple at x° . 
Moreover, for each e > 0, there is some to € -R>o such that tf < e for all i, and 
there are elements / 1; ...,/„ in A, such that (FT f ) (to) has rank n. 

Proof. Suppose that X\, . . . , X n are in Ala an d the vectors X\(x ), . . . , X n (x ) 
are linearly independent. Then, Xi(x), . . . , X n (x) are linearly independent for 
each x belonging to some open neighborhood Co of x Q in O. Thus, replacing 
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if necessary O by the subset Oq, and restricting all elements of A to Co, we 
assume without loss of generality that Ala{ x ) = K n for all x G O. 

Fix an e > 0. Let k be the largest possible integer for which there exist some 
fc-tuple (/i, . . . , fk), and some to G R>o with t° < e for all i, so that -F? * is 
defined on some neighborhood of 

[0,i?]x...x[0,i£] 

and with the property that (-F? f ) (t ) has rank fc. Note that k > 1, because 
there is some / G *4 with f(x°) ^ (otherwise, all brackets are zero, which 
would imply that ^4la( 2;0 ) = {0}), and, for this one-tuple, (Ff ) (0) = f{x°) 
has rank one. To prove the lemma, we need to show that k = n. 

Let (/i, . . . , fk), and io G -R> , all t® < e, be so that rank (F? r ) (io) = fc- 
By continuity of (F¥ , ) , we may assume without loss of generality that all 
the entries of to are positive. We pick a neighborhood W C IR™ of to, included 
in the domain _D? f , so that rank (_F? * ) (t) = k for all t in VF, and 

consider the slice M = Ff f : W — > C 

Claim: Every element of ^4 is tangent to M. Indeed, assume that this would 
not be the case. Then there would be some / G A so that f(M(s )) is not in 
the span of the columns of M*(so), for some s*o G W. Consider in that case the 
(k + l)-tuple 

(/l) • • • ,/fej/) j 

let G be the map Ff * ±, and take so ' ■= (so, 0) = (s°, . . . , s°, 0). Note that 
G is defined on a neighborhood of 

[0, S ?]x...x[0, S °]x{0}. 

We compute the Jacobian G»(so ')• Observe that 

G(t,t) = e tf M{t), 

so the Jacobian with respect to the variables t equals Q(t)M*(t), where Q(t) 
is the differential of the map x i— > e^X evaluated at M(i). In particular, at 
it ,i) = s ', one has M*(so). With respect to t, the derivative is f{e t ^M{t)), 
which evaluated at So' equals /(M(so)). We conclude that 

( F L,hj)Mo r ) = [M4so),f(M(s ))}, 

which has rank k + 1 (because the first k columns give a matrix of rank fc, since 
M is a slice, and the last column is not in the span of the rest). Since So ' has 
all its entries nonnegative and less than e, this contradicts the fact that k is 
maximal, and the claim is established. 

We proved in Lemma 4.1.8 that the set of all vector fields tangent to M 
is a Lie algebra. This set, as we just showed, contains A. So ^4lAj being the 
smallest Lie algebra containing A, must be a subset of the set of vector fields 
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tangent to M. Pick any t G W. Then, X(M(t)) must be in the column space 
of M*(t), for each X G -Ala- It follows that rankM*(£) = n, so necessarily 
k = n, as desired. I 



4.3 Accessibility Rank Condition 

We consider time-invariant continuous-time systems 

x = f(x,u), 

where states evolve in an open subset X C R", for some n. We assume that 
f{-,u) is C°° in x, for each u € IX. 

Remark 4.3.1 As done in the rest of the text, in order to be able to define and 
have reasonable properties for measurable controls, we suppose that the control- 
value set IX is a metric space and that the Jacobian of f(x, u) with respect to x 
is continuous on (x, u). However, for the basic accessibility result to be given, 
no regularity in u would be required if one would define "reachable sets" using 
only piecewise constant controls. □ 

Assume given a system x = f(x,u). We associate to it the following set of 
vector fields: 

A := {f u = f{-,u),ueU}. 

Definition 4.3.2 The Lie algebra of vector fields Ala is called the accessibil- 
ity Lie algebra associated to the system. The accessibility rank condition 

(ARC) at x° holds tfA LA {x°) = W 1 . □ 

An especially interesting class of systems, very common in applications, is 
that of control- affine systems, for which IX C K m , f(x,u) is affine in u: 

x = g {x) + uxgi(x) + . . . + u m g m {x) = g (x) + G{x)u (4.18) 

where gi, i = 0, . . . , m, are m + 1 vector fields and G = (</i, . . . , g m ), and, we 
assume here, 

e IX and linear span of IX = E m . (4.19) 

For systems of this general form, it is not necessary to use all the vector fields 
/„ when generating Ala'- 

Lemma 4.3.3 For a control-affine system, *4la = {<?o> • • • jSwiIla- 

Proof. It will suffice to show that the linear spans Lq of {go, . . . ,g m } and L^ 
of A = {f u = f(-,u),U € IX} are the same. Each element f u = go + ^2 u i9i of 
A is by definition a linear combination of the gi's, so it is clear that L_4 C Lq. 
Conversely, using u = shows that go € A, and thus also that G{x)u = U\gi + 
. . . + u m g m = f(x, u) — f{x, 0) s L.4 for every u G IX. To see that each c/j e L_a, 



4.3. Accessibility Rank Condition 155 

i G {1, . . . , m}, fix any such i, and write the fth canonical basis vector of W n 
as ei = Sip / /°j u ii f° r some Uj's in 11 and reals pj (recall that we are assuming 
that (4.19) holds). So, Qi = G(x)ei = J2j € j PjG(x)uj e L^. I 

In view of the characterization in Lemma 4.1.4, the accessibility rank condi- 
tion amounts to asking that there exist n vector fields X\ , . . . , X n , each of which 
can be expressed as an iterated Lie bracket [[. . . [f Ul , /« 2 ], /« 3 ], ••• , fu e ] (with 
possibly a different length i for each of the Xj's), so that X\{x ), . . . ,X n (x°) 
are linearly independent. In the case of control-affine systems, the condition 
means that there are n iterated brackets formed out of {go, . . . ,g m } which are 
linearly independent when evaluated at x°. 

Exercise 4.3.4 The length of the brackets needed in order to generate enough 
linearly independent vector fields (the "degree of nonholonomy" ) may be much 
larger than n. In other words, the ARC may hold at x°, yet its verification 
requires one to compute brackets of arbitrary length. Specifically, for each 
positive integer £ give an example of a system of dimension two, and a state x°, 
so that span („4 [J . . . {JAi-i)(x°) (recall Lemma 4.1.4) has dimension one but 
span („4 (J ■ ■ • U • / ^£){ x °) nas dimension two. □ 

Remark 4.3.5 Any time-invariant linear system x = Ax + Bu is control-affine, 
because it has the form (4.18) with go(x) = the linear vector field Ax and each 
gi(x) = the constant vector field bi (the ith column of B). To compute the Lie 
algebra ^4la for a linear system, we note that, in general, the Lie bracket of two 
constant vector fields is zero, and the Lie bracket of a constant and a linear vector 
field is a constant one: [h, Fx] = Fh. Also, [Ax, Ax] = 0. Therefore, an iterated 
bracket with I > 1, [[. . . [g il , g i2 ], g ia ], . . . , g ie \, is necessarily zero unless it has the 
form [[. . . [bi, Ax], Ax], . . . , Ax] = A l ~ l b t (or the form [[. . . [Ax, bi], Ax], . . . , Ax], 
which differs from it at most in its sign). Thus, the accessibility rank condition 
holds at a point x° if and only if 



n—li 



rank (Ax°, h, . . . , b m , Ab u . . . , Ab m , ■ ■ • , A n l b 1 ,...,A 

(as usual, the Cayley-Hamilton Theorem implies that we do not need to include 
any A bj with i > n). This is somewhat weaker than controllability, because of 
the additional vector Ax° , which does not appear in the Kalman controllability 
rank condition. For example, take the one-dimensional system x = x (with 
B = 0); we have that the accessibility rank condition holds at each x Q ^ 0, but 
the set of states reachable from such a state does not even contain the initial 
state in its interior (the reachable set from x° = 1, for instance, is the half-line 
[1, +oo)), much less is every other state reachable from x° . □ 

For each subset V C X, initial state x° € X, and time t > 0, we consider 
the set of states that are reachable from x°, in time exactly t, without leaving V 
(using measurable essentially bounded controls): 

K^ix ) := {z° | 3we£^(0,f) s.t. cj){s,0,x ,uj) e Wse [0,t] 

and 4>{t,0,x°,cu) = z } . 
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For each T > 0, we also consider the set of states reachable in time at most T , 
1Zy (x°) := Utefo t] ^-v{ x °)i When V = X, we drop the subscript. The whole 
reachable set from x° is H(x°) = \J t>0 TZ t (x°). For each V, z°, and t > 0, 
we also consider the set of states that are controllable to z° in time exactly t 
without leaving V, namely 

C£0°) := {x° | 3we^(0,t) s.t. <J)(s,0,x°,lu) G WsG [0,t] 

and (f){t,0,x o ,Lu) = z } , 

Cy (z ) := Utgfo t] Cy{x°), dropping subscripts if V = X, and C(z°) is the set 
of states that can be controlled to z , i.e., {J t>0 C t (x°). The main result can 
now be given. 

Theorem 9 Assume that the accessibility rank condition holds at x° . Then, 
for each neighborhood V of x° , and each T > 0, 

mtUy 1 \x°) ^ (4.20) 

and 

mtCy T {x") =£ 0. (4.21) 

In particular, TZ(x a ) and C(x ) have nonempty interiors. 

Proof. Pick a neighborhood V of x° and a T > 0. Since Ala{ x °) = K n , there 
is a finite subset of vector fields Ao Q A so that 

Ao LA (x°) = R n . 

By continuity of each e f f x on t and x, there exists some number e > with the 
property that, for all sequences of n elements fi, ■ ■ ■ , f n G Ao, and all sequences 
of positive numbers t\, . . . , t n with < U < e, 

£*,>/„ _ _ e *2/2gti/i a ,o ig (j ennec j anc j belongs to V . 

(To see this, one may first find a neighborhood Vi of x° and an e > so that 
e f fx G V for all < t < e, all x € Vi, and all the elements of „4o! then one 
finds a V2 so that e*^a; g Vi if i is small enough, / G „4o, and a; G V2, and 
so on inductively.) We pick such an e, and assume without loss of generality 
that £ < T/n. Lemma 4.2.8 then insures that there are vector fields f\ = 
/«i 1 ■ ■ ■ 1 fn = fu n belonging to Ao, and some to G ^> j such that < t® < e 
for all i and (Ff ? ) (to) has rank n. By continuity, we may assume that all 

t" > 0. By the Implicit Mapping Theorem, the image of -F? t , restricted to 
some subset of {(£1, . . . , t n ) \ < ti < e,i = 1, . . . , n}, contains an open set. 
Since 

Ff°,...j n (t) = e t '" f -...e' 2 W 1 ^ = <f>(t! + ... + t n ,0,x°, w) , 
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where us is the piecewise constant control having value U\ on [0, t\), value «2 on 
[ti, t\ + t2), • • • , and value u n on \t\ + . . . + t n —i, ti + . . . + t n ], this image is 
included in TZ^ (x ). 

To prove the statement for the sets Cy (x ), we argue as follows. Consider 
the "time-reversed" system x = —f(x,u). Since the accessibility Lie algebra of 
the time- reversed system is obtained from the vector fields { — f u ,u € 11} , and 
is, in particular, a subspace, it coincides with the accessibility Lie algebra of 
the original system. Thus, the accessibility rank condition holds for the time- 
reversed system. Pick any T and V. In general, if </>(T, 0, x,us) = z, then, for 
the reversed dynamics (j)~ we clearly have (j>~(T, 0, z, v) = x using the control 
v(t) := lu(T — t), (cf. Lemma 2.6.7), and, moreover, if the intermediate states 
4>(t, 0, x,us) are in V, then the same is true for the states <js~ (t, 0, z, v). Thus, the 
set Cy (z) for the original system coincides with 72- v (z) for the time- reversal, 
and the latter has a nonempty interior (by the first part of the proof) . I 

The property that 1Z(x°) have a nonempty interior is often called accessibil- 
ity (from x°). The accessibility rank condition is, in general, only a sufficient 
condition: 

Exercise 4.3.6 (a) Show that for systems of dimension one: x = f{x,v), X = 
K, the accessibility rank condition holds at a state x° if and only if 1Z(x°) has 
a nonempty interior, 
(b) Consider the following system with X = R 2 and IX = M 2 : 

X\ = U\ 

x 2 = <fi{xi)u 2 

where if : R — > R is a C°° function with the property that ip(z) ^ for each 
z jL but ^(0) = for all j = 0,1,2,... (for instance, <p(z) := er 1 /^ for 

z/ 0, f{0) '■= 0). Show that, for all x°, x° € int7\Ly (a; ), but, on the other 
hand, the accessibility rank condition does not hold at the points of the form 
(0,a;2). □ 

Exercise 4.3.7 Let x = A(t)x+B(t)u be a time-varying continuous time linear 
system, and assume that all the entries of A and B are smooth functions of t. 
Introduce the following system with state-space R" +1 : 

xo = 1 
x = A(xo)x + B(xq)u. 

Explain the relationship between the accessibility rank condition, applied to this 
system, and the Kalman-like condition for controllability of x = A(t)x + B(t)u 
studied in Corollary 3.5.18. □ 

The following easy fact is worth stating, for future reference. It says that 
the map x <— > <js(T, 0, x,us) is a local homeomorphism, for each fixed control us. 
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Lemma 4.3.8 Suppose that lo G £??(0,T) is admissible for x°, and let z° = 
4>(T, 0, x Q , u>). Then the map a = a^ : x i— > (/>(T, 0, a;,u;), which is defined on 
an open subset T> of X, is one-to-one, continuous, and has a continuous inverse. 
In particular, a(W) is a neighborhood of z°, for each neighborhood W of x° 
included in T>. 

Proof. We have that the control v(t) := u>(T — t) is admissible for z°, with 
respect to the reversed system x = —f(x,u). Moreover, if we write T>' for the 
set where f3(z) := 4>~{T, 0,z, v) is defined (the dynamics map for the reversed 
system), we have that ct{T>) = T>' and a o f3 and [3 o a are both the identity. 
From Theorem 1 we know that the domain T> is open and a is continuous, and, 
applied to the reversed system, also that T>' is open and (3 is continuous. I 



Reversible Systems 

In general, accessibility is weaker than controllability (as x = 1 illustrates). 
Sometimes, however, controllability can be characterized by the accessibility 
rank condition. 

Definition 4.3.9 A system is weakly reversible if^ is an equivalence rela- 
tion, and strongly reversible if for each x° and each u) € >C^ (0,T) admissible 
for x° , there is some v G C^(0,T) which is admissible for z° := (j){T, 0, x°,lo) 
and is such that 4>(t, 0, x°, lo) = <f>(T — t, 0, z° , v) for all t € [0, T] . □ 

That is, weakly reversible means that z° € TZ(x°) if and only if x° G 1Z(z ), 
and strongly reversible means that the same path that takes us from x° to z° 
can be traveled backward. Strong reversibility implies weak reversibility, but 
not conversely: the system 

-1 

1 

whose paths are circles transversed counterclockwise, is weakly reversible but 
not strongly so. 

A very interesting class of strongly reversible systems is obtained from those 
control- affine systems for which go = 0, that is, systems of the form 

x = wigi(x) + ... + u m g m (x) = G{x)u. (4.22) 

We assume that (4.19) holds and also that IX is symmetric: u € 11 => — u € IX. 
These are called systems without drift (because, when u = 0, the state does not 
"drift" but, instead, remains constant) and they arise as kinematic models of 
mechanical systems. For such systems, we may let i>{t) := —lo{T — t) in the 
definition of strongly reversible system, and verify the property using the fact 
that x = G{x)lu implies (d/dt)x(T - t) = G{x{T - t))u(t). 
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Remark 4.3.10 It is worth noticing that the first-order test for local con- 
trollability provides no information in the case of systems without drift. The 
linearization of (4.22) at any state x° and input value is x = G*(x )u. This 
system cannot be controllable, unless m > n. □ 

Proposition 4.3.11 Assume that the accessibility rank condition holds at x°. 
If the system x = f(x, u) is weakly reversible, then 

x° G int (n{x°)f]C{x )^ 

Moreover, if it is strongly reversible, then 

*° G int(7ef(* )nCvV)). 

for every neighborhood V of x° and every T > 0. 

Proof. We know from Theorem 9 that TZ(x°) has a nonempty interior. Pick 
any state z° in the interior of TZ(x°). Because of weak reversibility, there is 
some control ui G £^(0,T), for some T > 0, such that <p(T, 0, z , uj) = x a . By 
Lemma 4.3.8, there is an open subset W of lZ(x°) which contains z° and is such 
that W' := <j>(T, 0,W,u>) is an open subset containing a; . But W C TZ(x°), 
since x G W' implies x = <f>(T, 0, z, u>) for some z G W C !Z(x°). The proof for 
C(x°) is analogous. 

Assume now that strong reversibility holds. Pick To and V, and let T := 
To/2. Again by the Theorem, we know that TZy (x°) has a nonempty interior. 
Pick any z° in the interior of TZy (x°), and let < T\ < T and u) G £^(0,Ti) 
be such that 0(Ti, 0, x" , lo) = z" and <j)(t,0, X°,u>) G V for all t G [0,Ti]. Let 
v G £^ (0,Ti) be as in the definition of strong reversibility, for this u>. By 
the continuity of z i— > ^(-,0,2, f) (as a map into C ([0, Ti], X)), there is an 
open set W containing z° and such that <p{t, 0, z,w) G V for all z G W and all 
t G [0,Ti], and we may assume without loss of generality that W C 1Z^ (x°). 
Moreover, by Lemma 4.3.8, the image W := 4>(T\, 0, W, v) is an open subset 
which contains x°. Note that W" C int7?.y °(a; ). Indeed, for each x G W 



,<T , 



there is, by definition, an element 2; of TZ V (x°) which is steered to x in time 
< T and without leaving V (in fact, using the control v); and, in turn, z can be 
reached from x° in time T\ without leaving V. The proof for C^ (x a ) is entirely 
analogous. I 

Corollary 4.3.12 For a weakly reversible system, if the accessibility rank con- 
dition holds at every state x G X and X is connected, then the system is com- 
pletely controllable. 

Proof. Pick any state x°. Pick any z° G TZ(x°). From Proposition 4.3.11, we 
know that z° G int 7?.(;2 ); by transitivity of reachability, !Z(z°) C TZ(x°), so this 
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proves that 1Z(x°) is open. Now pick z° lZ(x a ). Again from the Proposition, 
z° G intC(z°). But C(z ) f] TZ(x°) = (again, by transitivity of reachability) . 
Thus both the reachable set TZ(x°) and its complement are open; since X is 
connected and TZ(x°) ^ 0, TZ(x°) = X, as wanted. I 

Example 4.3.13 (Nelson's car.) The following is a simplified model of a front- 
wheel drive automobile. The state space is K 4 , and the coordinates of the state 





denote, respectively, the position of the center of the front axle (coordinates 
Xi, x 2 ), the orientation of the car (angle tp, measured counterclockwise from the 
positive x-axis), and the angle of the front wheels relative to the orientation of 
the car (9, also counterclockwise); see Figure 4.1. 

X2 




X\ 



Figure 4.1: ^-dimensional car model. 



We now sketch how to derive the equations of motion. There are two con- 
straints to take into account, corresponding respectively to the requirement that 
the front and back wheels do not slip (no sideways motion of tires; the only mo- 
tions allowed are rolling and rotation in place). The front wheels are parallel to 
the vector (cos((9 + </j),sin(0 + 93)), so the instantaneous direction of movement 
of the center of the front axle is parallel to this vector: 

'' «2(*) i :::r a ; :< j (4.23) 



(ii 



cos(t) 
sin(8 - 
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for some time-dependent scalar U2(t). Observe that xi(t) 2 + X2{t) 2 = U2(t) 2 , so 
«2 is the velocity of the car, which we think of as a control (engine speed). The 
center of the rear axle has coordinates {x\ — £cos <p, X2 — isinip)' , where £ is the 
distance between the two axles. The velocity of this point must be parallel to 
the orientation (cos ip, sin ip)' of the back wheels: 

d d 

simp— (xi —£ cos ip) — cos tp— (X2 — £ simp) = 0. 

Substituting (4.23) into this orthogonality relation, and after performing some 
trigonometric simplifications, we obtain: 

£ip = «2 sin 9 . 

To avoid extra notation, we assume from now on that £ = 1. The angle 9 
changes depending on the steering wheel position; we take the velocity at which 
the steering wheel is being turned as another control, u\. In summary, with 
the control- value space IX = K 2 , and writing controls as u = (1*1,1*2)' € R 2 , we 
obtain the following system: 



(4.24) 



Note that a control such that U2{t) = corresponds to a pure steering move, 
while one with U\(t) = models a pure driving move in which the steering wheel 
is fixed in one position. In general, a control is a function u{t) which indicates, 
at each time t, the current steering velocity and engine speed. 

In practice, of course, the angle 9 would be restricted to some maximal 
interval (— 9o,9q); thus we could take the state space as R 3 x (— 9q,9q), which 
would not change anything to follow. More importantly, the orientation angle 
ip only makes sense modulo 2ir, that is, angles differing by 2ir correspond to 
the same physical orientation. Nonlinear control theory is usually developed in 
far more generality than here, allowing the state space X to be a differentiable 
manifold; thus, a more natural state space than M 4 would be, for this example, 
R 2 x 8 1 x (— #o, #o), that is, the angle tp is thought of as an element of the unit 
circle. 

The model is control- affine; in fact, it is a system without drift x = uigi + 
u 2 <?2j where the defining vector fields are g\ = steer and g 2 = drive: 

cos(ip+ 9)' 

a - II II I Sin (^+ 6 ') 

fa - I ,. ! • fh- I! s - md 

By Proposition 4.3.11 and Corollary 4.3.12, we will have complete controllability, 
as well as the property that all states near a given state can be attained with 
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small excursions and in small time, once we prove that the accessibility rank 
condition holds at every point. (Of course, it is quite obvious from physical 
reasoning, for this example, that complete controllability holds.) We compute 
some brackets: 



— sin(<£ + 9) ' 

cos(y + 9) 

cos 






wriggle := [steer, drive] = 

and 

slide := [wriggle, drive] 



(The bracket [wriggle, steer] equals drive, so it is redundant, in so far as check- 
ing the accessibility rank condition is concerned.) It turns out that these four 
brackets are enough to satisfy the accessibility test. Indeed, one computes 

det (steer, drive, wriggle, slide) = 1, 

so the condition indeed holds. 

It is instructive to consider, in particular, the problem of accessibility starting 
from the special state x° = 0. We may interpret this state as corresponding to 
a "parallel parked" car; accessibility allows us to move in various directions 
without leaving a small neighborhood, that is, without hitting the cars in front 
and behind. For <p = 9 = 0, wriggle is the vector (0, 1, 1,0), a mix of sliding in 
the x 2 direction and a rotation, and slide is the vector (0, 1,0,0) corresponding 
to sliding in the x 2 direction. This means that one can in principle implement 
infinitesimally both of the above motions. The "wriggling" motion is, based on 
the characterization in Exercise 4.2.6, the one that arises, in a limiting sense, 
from fast repetitions of the following sequence of four basic actions: 

steer - drive - reverse steer - reverse drive. (*) 

This is, essentially, what one does in order to get out of a tight parking space. 
Interestingly enough, one could also approximate the pure sliding motion in the 
x 2 direction: u wriggle, drive, reverse wriggle, reverse drive, repeat' corresponds 
to the last vector field, "slide", which at x = coincides with this motion. 
Note that the square roots in Equation (4.17) explain why many iterations of 
basic motions (*) are required in order to obtain a displacement in the wriggling 
direction: the order of magnitude t of a displacement is much smaller than the 
total time needed to execute the maneuver, 4\ri. □ 

Exercise 4.3.14 Consider the following three systems, all with state space R 3 
and control- value space R 2 : 



Xl = Ml, 


X 2 = 


= u 2 , 


X3 = X 2 U\ — XlM 2 


(Si) 


X\ = Ml, 


X 2 = 


= u 2 , 


X 3 = X 2 Ui 


(s 2 ) 


Xl = Ml, 


X 2 = 


= u 2 , 


±3 = X 2 Ul + XlU 2 . 


(Es) 
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Answer (justify!): which of these systems are controllable? (Hint: For (S3), 
you may want to analyze the function i(xi,X2,x 3 ) := £1X2 — x 3 .) For related 
material, cf. Exercises 4.3.16 and 9.4.2, and Examples 5.9.16 and 9.2.14. □ 

Exercise 4.3.15 Consider a rigid body which is being controlled by means of 
one or more applied torques (for example, a satellite in space, under the action 
of one or more pairs of opposing thruster jets) . We only study here the effect of 
controls on angular momenta; more complete models incorporating orientations 
and even linear displacements are of course also possible. With the components 
of x = (xi,X2,X 3 )' denoting the angular velocity coordinates with respect to 
the principal axes, and the positive numbers I\ , I2 , I 3 denoting the respective 
principal moments of inertia, this is a system with X = R 3 , U = K m , where m 
is the number of torques; the evolution is represented by (the Euler equations 
for rotational movement): 

Ix = S{x)Ix + Tu, (4.25) 

where / is the diagonal matrix with entries I\, 12,13 and where T is a matrix 
whose columns describe the axes along which the torques act. The matrix S(x) 
is the rotation matrix 

S(x) = 

(Equivalently, the equations can be written as Ix = I x x + Tu, where "x" is 
the vector product in R 3 .) 

(a) Consider the case in which there are two torques (m = 2), which act about 
the first two principal axes, that is, T 6 M 3x2 has columns (1, 0, 0)' and (0, 1, 0)'. 
The equations can be written as follows: 

X\ = 0,1X2X3 + biUi 
x 2 = a 2 xix 3 + b 2 u 2 
X3 = a 3 x 1 x 2 

where 01 = (h ~ h)/h, &2 = (h ~ h)/h, and a 3 = (h - h)/h, and 61,62 are 
both nonzero. You may assume that b\ = 62 = 1- Show that the accessibility 
rank condition holds at every point, if and only if I\ ^ 12- 

(b) Now consider the case in which there is only one torque, acting about a 
mixed axis. Taking for simplicity the case in which there is rotational symmetry, 
Ii = I2, the equations can be written as follows: 




±1 


= 


ax2X 3 + b\u 


X 2 


= 


—ax\X 3 + &2« 


x 3 


= 


b 3 u 



where we assume b/0, and the 6j's are real numbers. Show that the accessi- 
bility rank condition holds at every point of the state space if and only if b 3 ^ 
and b\ + b\ ^ 0. □ 
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Exercise 4.3.16 Consider a model for the "shopping cart" shown in Figure 4.2 
("knife-edge" or "unicycle" are other names for this example). The state is 
given by the orientation 9, together with the coordinates X\, x 2 of the midpoint 
between the back wheels. 

x 2 




Figure 4.2: Shopping cart. 



The front wheel is a castor, free to rotate. There is a non-slipping constraint 
on movement: the velocity {x\,x 2 )' must be parallel to the vector (cos 9, sin 9)' . 
This leads to the following equations: 

x\ = uiCOs9 
x 2 = u\ sin 9 
6 = u 2 

where we may view U\ as a "drive" command and u 2 as a steering control (in 
practice, we implement these controls by means of differential forces on the two 
back corners of the cart). We view the system as having state space K 3 (a more 
accurate state space would be the manifold R 2 x S 1 ). 

(a) Show that the system is completely controllable. 

(b) Consider these new variables: Z\ := 9, z 2 := a;icos# + x 2 sin9, z^ := 
Xism9 — x 2 cos9, Vi := u 2 , and v 2 := U\ — u 2 z%. (Such a change of variables 
is called a "feedback transformation" .) Write the system in these variables, as 
z = f(z, v). Note that this is one of the systems £; in Exercise 4.3.14. Explain 
why controllability can then be deduced from what you already concluded in 
that previous exercise. □ 
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The converse of Theorem 9 (p. 156) is not true: it may well be that the accessi- 
bility rank condition does not hold at a state x , yet TZ^ (x°) has a nonempty 
interior for each neighborhood V of x° and each T > 0; cf. Exercise 4.3.6. In 
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order to provide a partial converse of the Theorem, we first need to develop a 
few additional facts regarding flows and Lie brackets. 

The "Ad" Operator 

Let X G V(O) and pick any (t, x°) in T>x (the domain of definition of the flow 
of X). If Y is a vector field defined on an open subset Oy of O which contains 
e tx x°, we define 



Ad tx Y{x°) := (e- tx ) it (e tx x )-Y(e tx x ) 



(this is a matrix, the Jacobian of e~ tx evaluated at e tx x°, multiplied by the 
vector Y(e tx x°); we will sometimes use dots, as here, to make formulas more 
readable). Geometrically, the vector Ad t xY(x°) is the "pull-back" to a; , along 
the flow of X , of the vector Y(e tx x°). For example, let, in R 2 , 



-1 

1 



Y(x) := 



Xl 

-xi + x\ 



and X(x) = Ax. Then, 

Ad^ x Y(x°) 



= (-tt/2) j 4 



(a clockwise rotation by 7r/2 of Y(e x )), see Figure 4.3 




IT 

2 ' 



Figure 4.3: Example of Ad operator. 

The state e~ tx e sY e tx x° = <fix{—t, 4>y{s, <f>x{t, x ))) is well-defined provided 
that s is near zero (because e~ tx z° is defined for z° = e tx x° and, thus, also for 
all z° sufficiently near e tx x° , and e sY e tx x° is defined, and is close to e tx x°, 
when s is small), and 



Ad tx Y(x°) 



i) 



e -tX e sY e tX x _ 
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We may view Ad t xY as a (smooth) vector field on the open subset O C O 
consisting of all x° such that (t,X°) G T> x and e tx x° € Oy. Given a third 
vector field Z € V(C), and (s,a;°) € D^, we may consider Ad s ^(Adtx^)(2: )» 
provided e sZ x° E O (that is, if (t,e sZ x°) E V x and e tx e sZ x a £ O y )\ we 
write this simply as Ad s zAd t xY(x°). Similarly, one may consider iterations of 
applications of the Ad operator. The following is an easy consequence of the 
chain rule. 

Lemma/Exercise 4.4.1 Let X x , . . . , X k , Y e V(O). Then, 

Ad tkXk ...Ad tlXl Y(x°) = 

(e~ tkXk . . . e"' 1 ^)^ (e tlXl . . . e tkXk x°) ■ Y (e tlXl . . . e tkXk x°) (4.26) 

if (t k , x°) e V Xk , (t fc _i, e tkX "x°) e T> Xk _^ ..., (ti, e' 2 * 2 . . . e tkXk x°) e V Xl . In 
particular, 

Ad tkX ...Ad tlX Y(x°) = Ad (tl+ ... +tfc)x y(z ) 

when all the Xi equal the same X. □ 

We may view Ad t x as an operator on vector fields (defined on open subsets 
of O). The following fact relates "Ad" to the "ad" operator introduced earlier, 
in Equation (4.4): 

Lemma 4.4.2 Let X e V(0) and {t°,x°) e V x . Pick any Y € V(O y ) such 
that e*° x x° £ Oy- Then, 

dAd tx Y(x°) 



01 



= Ad t o x [X,Y](x°). 
t=t° 

Proof. We have, for all (t,x) € ~D X , e~ tx e tx x = x, so taking d/dx there 
results 

(e- tx )^ (e tx x) ■ (e tx )^(x) = I 

and, taking d/dt, 



(4.27) 
On the other hand, 



-((e tx ) (x)] = -—(e tx x) = — -(e tx x) 

= ^-X(e tx x) = X*(e tx x)-(e tx )«(x), 

Ox ' * 

so substituting in (4.27) and postmultiplying by y(e tx ) (x)j there results the 
identity: 



d 

m 



((e"«). (e tx x) ) = -(e-«). (e«x) ■ X, (e* x x) . (4.28) 



4.4. Ad, Distributions, and Frobenius' Theorem 167 

Thus, 



d_ 
Ft' 



;((e-«) t (eS).y(e«,)) 

= _ ( e -«) ^ (e tx x) ■ X, (e tx x) ■ Y (e tx x) 

+ (e~ tx )^ (e tx x) ■ Y, (e tx x) ■ X (e tx x) 

= (e- tx )^e tx x)-[X,Y]\e tx x), 

as required. I 

A generalization to higher-order derivatives is as follows; it can be inter- 
preted, in a formal sense, as saying that Ad t x = e x . For each k = 0, 1, . . ., 
and each vector fields X, Y, we denote adj^y = Y and adj^ Y := a.dx{&<i x Y). 

Lemma 4.4.3 Let X £ V(O), {t°,x°) £ V x , Oy Q O, and Y £ Y{Oy) be so 
that e t0x x° £ Oy. Then, for each k = 0, 1, 2, . . ., 



d k Ad tx Y(x° 



dt k 



AA tax& d x Y(x°) 



t=t° 
In particular, for any X,Y £ V(O) and each x° £ O 

d k Ad tx Y{x° 



dt k 
for each k. 



a,d x Y{x°) 



t=o 



Proof. Fix the state x° and consider the function j(t) := Ad tx Y(x°); we 
want to prove that j( k >(t) = Adt X a.d x Y(x°) for each k and for all t such that 
(t, x°) £ T> x and e tx x° £ Oy. We proceed by induction on k. The case k = 
is trivial, since, by definition, ad^y = Y. Now, assume that the conclusion has 
been shown for k. Then, 

7 (fc+1) (*) = j t l {k \t) = ±Ad tx Z(x°), 

where Z is ad x Y, seen as a vector field defined on Oy. By Lemma 4.4.2 (the 
case k = 1), ^ k+1 \t) = Ad tx a,d x Z{x°) = Ad tx ad^- +1 y(a; ), as desired. 

In particular, consider any X,Y £ V(O) and x° £ O. As (0,x ) £ T> Xj the 
conclusions hold for t = 0. I 

The next few technical results will be needed later, when we specialize our 
study to analytic systems. 

Lemma 4.4.4 Suppose X and Y are analytic vector fields defined on O. For 
any x° £ O, let 1 = T X , X Q := {t £ R | (t,X°) £ V x }. Then, the function 

7 : 1 -> R" : t i-> Ad tx y(i°) is analytic. 
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Proof. Let a{t) := (e~ tx ) t (e tx x°) , seen as a function I -» R nxn . Note that 
a(0) = /, and that, by Equation (4.28), the vector (e tx x°,a(tj) is the solution 
of the differential equation 

x{t) = X{x{t)) x(0) = x° 

a(t) = -a(t) ■ X„(x(t)) a(0)=I. 

This is a differential equation with analytic right-hand side, so x(-) and a(-) are 
both analytic (see, for instance, Proposition C.3.12). Then, "/(t) = a(t)Y (x(t)) 
is also analytic. I 

Proposition 4.4.5 If the vector fields Y\, . . . , Yi and X are analytic, then 

span {Ad t xYj(x°), j = 1, . . .£, t e l x , x °} 
equals 

span lad k x Yj(x°), j = 1, . . .1, k > o] 

for each x° G O. 

Proof. Fix x a € O. Let So and 5*1 be, respectively, the sets of vectors v € E" 
and n g R" such that 

z/'ad^y^a; ) = 0, j = 1, . . . t , fc > (4.29) 

and 

V' AdtxYj(x ) = 0, j = l,..J,teI„. (4.30) 

Take any v G S - For each j = 1, . . . £, by Lemma 4.4.3, 

, a fe Ad t x^(x°) 



9i fe 



= 0, Vfc > 0. 
t=o 



Since, by Lemma AAA, AdtxYj(x°) is analytic as a function oft, this means that 
u'AdtxYj{x°) = 0, so i/ e Si. Conversely, if v € Si, then u / Ad tx Y j (x°) = 
implies that all derivatives at zero vanish, so v € Sq (analyticity is not needed 
here). Thus So = Si, and the result is proved. I 

Corollary 4.4.6 Let the vector fields Yi, . . . , Yg, and X be analytic, and pick 
any (t,x°) G T>x- Let 

d := dim span {Yj{e tx x°),j = !,...(] . 

Then, 

d < dimspan l&d x Yj{x ), j = !,...£, k > o] . 

Proof. Since Ad tx Yj{x°) = Q-Y 3 (e tx x°), where Q := (e - ^)^ (e tx x a ) is a 
nonsingular matrix, d = dimspan {AdtxYj(x°),j = 1, . . .£}. The result follows 
then from Proposition 4.4.5. I 

Exercise A.A.I Show that Corollary 4.4.6 need not hold for smooth but non- 
analytic vector fields. □ 
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Distributions 

A distribution (in the sense of differential geometry) is the assignment of a 
subspace of tangent vectors to each point in the state space. 

Definition 4.4.8 A distribution on the open subset O C W 1 is a map A 
which assigns, to each x £ O, a subspace A{x) ofW 1 . A vector field f £ V(O) 
ispointwise in A, denoted f £ p A, if f{x) € A(x) for allx £ O. A distribution 
is invariant under a vector field f £ V(O) if 

g£ p A => [f,g]€ p A, 

and it is involutive if it is invariant under all f G p A, that is, it is pointwise 
closed under Lie brackets: 

f e p A and g £ p A =4- [f,g] £ p A . 

The distribution generated by a set of vector fields /i, . . . , f r G V(O) is defined 
by 

A fu ...j r (x) := span{/i(x),...,/ r (cc)} 

for each x £ O. A distribution has constant rank r z/dimA(:r) = r for all 
x £ O. □ 

Lemma 4.4.9 Suppose that A = Ay li j r is a distribution of constant rank r. 
Then, 

1. The following two properties are equivalent, for any / £ V(O): 

(a) / £ p A 

(b) For each x° £ O, there are a neighborhood Oq of a; and r smooth 
functions on : Oq — » R, i = 1, . . . ,r, so that 



f{x) = Y,^( X )M X ) for a11 x£O a . (4.31) 



2. The following two properties are equivalent, for any / £ V(O): 

(a) A is invariant under /. 

(b) [/, fj] £ p A for each j £{!,..., r}. 

3. Finally, the following two properties are equivalent: 

(a) A is involutive. 

(b) &,&]€, A for e31i,j e{l,..., r}. 
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Proof. The equivalence of la and lb was already established in Corollary 4.1.6 
(applied with W = O, p = r, and Q the matrix function whose columns are the 
fi'B). 

The implication 2a => 2b is clear, since each fj G p A. To prove the converse, 
pick any j G p A, and consider any point x G O. By the first equivalence, we 
can write g{x) = Y^ r j=i a j( x )fj{ x )i on some neighborhood Oq of x , for some 
smooth functions ay's. Lemma 4.1.2 gives, then, as functions on Oq' 

r 

if, g] = E^[/'^] + ( L /^)^- 

Since fj(x°) G A(x°) and [/, fj](x°) G A(a; ) for all j, we conclude that 
[f,g}(x°)eA(x°). 

Similarly, the implication 3a =^ 3b is clear, since each fi G p A. To prove 
the converse, we need to show that A is invariant under any / G p A. The 
previous equivalence tells us that it is enough to check that 2b holds, for each 
such /. Since each A(x) is a subspace, however, this is the same as asking that 
[fij /] £ p A for all i and all / G p A, that is, the property that A is invariant 
under /$; but again by the previous equivalence, this holds if [fi, fj] G p A for 
all j, which was the assumption. I 

Exercise 4.4.10 Provide examples of distributions A.f lt ...,f r (necessarily not 
having constant rank) for which each of the above equivalences fail. □ 

Exercise 4.4.11 A distribution is said to be smooth if it is locally generated by 
sets (possibly infinite) of vector fields, that is, for each x° G O there is a subset 
T C V(O), and there is an open subset Oq C O which contains x°, such that, 
for each x G Co, A(x) is the span of the vectors {f(x), f € J 7 }. Show that, if A 
is a smooth distribution of constant rank r, then for each x° G O there is some 
open subset Oq C O which contains a; , and a set of r vector fields fi, . ■ ■ , f r , 
such that A = Af lt j r on O , that is, A(x) = spanj/^a;), . . . , f r (x)} for each 
xeO - □ 

Invariance of a distribution under / is equivalent to invariance under the 
linear operators Adt/, in the sense that A is invariant under / if and only if 
Ad t fg(x) G A(x) for each (t,x) G T>f and each g G p A. This fact, proved next, 
is perhaps the most important property of the Lie bracket operation. (We need 
a slightly stronger version, in which g is not necessarily defined globally.) 

Lemma 4.4.12 Suppose that A = Af lt .„j r has constant rank r, and let X G 
V(O). Then, the following two properties are equivalent: 

1. A is invariant under X. 

2. Let 0\ be an open subset of O and let t G M be so that (£, x) G T>x for 
all leOj. Define O := e tx 1 . Assume that Y G V(0 O ) is such that 
Y(z) G A(z) for each z G O a . Then, Ad tx Y(x) G A(x) for each x G 0\. 
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Proof. We first prove that 2 => 1. Pick any Y G p A, and any x° G O. Take 
any open subset 0\ C O containing x° and any e > so that (£, x) G Djf for all 
\t\ < e and all x G 0\. It follows from 2 that Adtx^(a;) £ A(cc) for all i near 
zero and all i £ Oi, so, in particular, Ad t xY(x°) G A(x ) for small t. Then, 
since A (a; ) is a subspace, also 

i (Ad 4X y(x°) - Y(x )) e A(x°) 

and, therefore, since A(x°) is closed, the limit [X, Y](x ) of this expression (cf. 
Lemma 4.4.2) is in A(a; ). So A is invariant under X. 

We now prove the converse. We first show that, for each given x G O, and 
letting X := {t G R | (i, a;) G ©x}, 

a»(t) := Ad t x/i(x) G A(x) for each i G {1, ... ,r} and each t G X. (4.32) 

For this, it is enough to prove that, for each vector v such that v'fj(x) = for 
all j = 1, . . . , r, necessarily z^'a^ = for all i. So pick such a z/, and consider 
the row vector function pit) := v' ' M(t), where M{t) := {a\(t), . . . , a r (t)). Note 

thatp(0) = i/(/i(aO,...,/ r (aO)=0. 

Assume that it is not the case that p = 0. Suppose there is some t > so 
that p{t) ^ (if this only happens for some t < 0, the argument is analogous). 
Let 

t := inf{£ > 0,t€l,p{t) ^ 0}. 

Note that p{to) = 0. Consider the point z := e toX x. For each i = l,...,r, 
since [X, fi] G p A (invariance under X), from Lemma 4.4.9 we know that 
there is a neighborhood Oq of z and there exists a set of smooth functions 
{ 7ij , j = 1, . . . , r} C F(O ) such that 

r 

[*,£](*) = $^Tfc(*)/j(*), forallzGOo- (4.33) 

(We may assume that the neighborhood O is common to all i.) By Lemma 4.4.2 
and Equation (4.33), 

di(t) - AdtxlXJ,} (x) = (e~ tx )^ (e tx x) ■ [X,f t ] (e tx x) 
jij (e tx x) a ^t) 






as long as |i — to I is small enough so that e tx x G Co- So we have that 

M{t) = M{t)T{t), 

where we are denoting r(t) := ijjiie tx xj). Then, p(t) = p(t)r(t), andp(t ) = 0; 
by uniqueness of solutions of p = pT we have that p = for all t sufficiently 
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near to. This contradicts the definition of to- So it must be that p = 0. This 
concludes the proof of (4.32). 

Finally, pick any Oq, Oi, t, and Y as in 2, and any x Q £ 0\. Taking if 
necessary a smaller neighborhood 0\ of x° , we know that there are smooth 
functions {7^, i = 1, . . . , r} C F(O ) such that V = X^I=i 7*/i on ^o- Therefore, 
letting a.i(t) = Adtx fi{% ) for each i, 

Ad tx Y(x°) = (e- tx )Je tx x°)-Y(e tx x°) = £ 7i (e*** ) <*i(t) , 

which is in A(x ) because each Oj(t) € A(a; ) by (4.32). I 

By induction, we also have then the following consequence of Lemma 4.4.12 
(see Figure 4.4). 




3 2 "1 

Figure 4.4: Z = Ad t3X3 Ad t2 x 2 Ad tlXl Y . 







Corollary 4.4.13 Suppose that A = A.f lt „.j r has constant rank r and is in- 
variant under each of X\, . . . , Xk £ V(O). Let Ok be an open subset of O and 
let S\, . . . , Sk be real numbers with the following property: 



(si,x)£V Xi , Vx £ Oi, i = l,...,k, 



where we define 



O 



fe-i 



„s k X h 



O k . 



, On-- 



^s 2 X 2 



e~~ 2 O 2 ,O := e s ^0 1 



Assume that Y £ V(O ) is such that Y(z) £ A(z) for each z £ O - Then, 
Ad SkXk ...Ad SlXl Y{x) £A(x) forallxeOfc. D 

Definition 4.4.14 A distribution A of constant rank r is completely inte- 

grable if, for each x° £ O there exists some neighborhood Oq of x° and a 
smooth function 

* : Co -> K"~ r 

such that 

for all x £ Oq . 



ker**(a;) = A(x) 



(4.34) 

D 
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Equation (4.34) says that ty*(x)v = if and only if v € A(x). Since A 
is assumed to have constant rank r, this implies that the Jacobian ^*(a;) has 
constant rank n — r. If A = A^ ,...,/ r , then, since {fi(x), . . . , f r {x)} is a basis 
of A(cc) for each x, we have that Equation (4.34) is equivalent to: 

V*(x)fj{x) = for j = l,...,r (4.35) 

together with: 

rank ty* [x) = n — r . (4.36) 

In other words, if we consider the rows ipk, k = 1, . . . , n — r, of Vl/, we have that 

L fj ij} k {x) = V-ip k {x)fj{x) = for k = l,...,n-r, j = l,...,r, (4.37) 

and: 

{\7tpi(x), . . . , Vipn-r{x)} are linearly independent. (4.38) 

Therefore, complete integrability of A means that, locally about each point, the 
set of r first order linear partial differential equations 

Vil>{x)fi{x) =0j = l r 

has n — r independent (in the sense of Jacobians being independent) solutions. 

Remark 4.4.15 The terminology arises as follows. A (smooth) function tp : 
O — * M. is an integral of motion for the flow induced by a vector field / if ip(x(t)) 

is constant along solutions. This means that ^ t — = for all x° and t, 

which is equivalent to the statement that Vip(x)f(x) = 0. Thus, complete 
integrability means that, locally, we can find n — r independent integrals of 
motion. In geometric terms, since the level sets of ^f(x) = c are differentiable 
submanifolds of dimension r (because ^* has constant rank n — r) complete 
integrability means that the space can be locally partitioned ( "foliated" ) into 
submanifolds of dimension r, in such a manner that the vector fields f\ are 
tangent to these submanifolds. □ 

Frobenius' Theorem 

The following construction provides the main step in the proof of Frobenius' 
Theorem. We state it separately, for future reference. 

Lemma 4.4.16 Suppose that A = Af lt ... t f r is involutive of constant rank r. 
Then, for each x Q G O there exists some open subset Oq containing x , an e > 0, 
and a diffeomorphism II : Oq — > (— £,e) n , so that the following property holds. 
If we partition 

n=(j£), n 2 : O ^(-e,e) n - r , 
then A (a;) = ker (n 2 ) t (a;) for all x £ O . 
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Proof. Pick x° G O. Let w r +i, • • • , v n be vectors so that 

{fi(x°), • • • , fr{x"), v r+1 , ...,v n } 

is a basis of R™. Introduce the vector fields fi = Vj, for i = r + l,...,n. 
Then {/i(a;), . . . , / n (a;)} is a basis of 1™ for all x in some open subset C 0. 
Replacing O by O, we assume from now on that {fi(x), . . . , f n (x)} are linearly 
independent for all x G C We consider the following map: 

M : (-£, e) n -> O : t = (ti, . . . , i„) i-> e 4l/l . . . e w "a; , 

where e is chosen small enough so that, for all j > 1, (tj-i, e*" 3 ' . . . e* r *^"x°) G 
'Df_ 1 whenever t G (— £, e) n . We have, for each j = 1, . . . , n: 

|^(t) = (e* 1 * . . . e *i-i/i-i) t ( e *^ . . . e'"-^ ) • /,- (e*^ . . . e*"'"* ) 

= Ad ( _ tl)/l ...Ad ( _ t ,_ l)/ ,_ 1 / i (M(t)) 

(the last equality because e* J ^ . . . e tn ^ n x° = e - ' 3-1 ^- 1 . . . e _ ' 1 -' ?1 M(t), and ap- 
plying (4.26)). In particular, 



M*(0,...,0) = (^(x ),...,^^ )) . (4.39) 



We claim that 
dM 



dtj 



(t) G A(M(t)), for each j = l,...,r and each t G (-£,£)" . (4.40) 



To show this for each j, we apply Corollary 4.4.13, with k = j — 1, Y = fj, 
Xi = fj-i, . . . , X k = /i, si = -tj-i, ..., Sk = ~h, and O k is chosen as any 
small enough neighborhood of M(t) with the property that e - ^ -1 ^ -1 . . . e~ tl ^ 1 x 
is defined for every x G O k . Involutivity of A insures that A is invariant under 
all the Xj's. As Y G p A, we conclude from that Corollary that 

Ad(_ tl)/l ...Ad(_ t ._ 1 ) / ._ 1 / j (2) e A(z) 

for each z G Ok, and, so, in particular, for M(t). This proves the claim. 

Because of (4.39), by the Inverse Function Theorem, taking a smaller e 
if necessary, and letting O be the image of M , we may assume that M is a 
diffeomorphism, that is, there is a smooth function II : O — > (—£,£)" such that 
II o M(t) = t, for all t G (— £,£)"• In particular, 

n»(M(t)) M*(t) = / VtG(-£,e)™. (4.41) 

We partition II as I, where II2 : — » (— £,£)™~ r . Then (4.41) implies: 

\ ii 2/ 

n 2 »(M(t))— (t) = VtG (-£,£)", j = l,...,r. (4.42) 
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Since the vectors ^M-(t) are linearly independent (because of (4.41)) and they 
are in A(M(t)) (because of (4.40)), they form a basis of A(M(t)); thus 

II 2 .(M(t)) A(M(t)) = 0, 

and, since rank n 2 *(M(t)) = n-r, A(M(t)) = kern 2 *(M(t)). Each point of O 
is of the form M(t), for some t G (— £,e)™, so this means that A(x) = kern 2 *(:z:) 
for all x G O, as wanted. I 

Theorem 10 (Frobenius) Suppose that A = Af lt ...,f T has constant rank r. 
Then, A is completely integrable if and only if it is involutive. 

Proof. Sufficiency follows from Lemma 4.4.16, taking ty := II2. The necessity 
part is easy: Pick any x° G O, let Oq and ^ be as in the definition of integra- 
bility, and consider the rows ipk, k = 1, . . . ,n — r, of *P, so that (4.37) holds. 
Thus, for each i, j G {1, . . . , r}, 

hfiJ^kix) = L fi L fj i; k (x) - L fj L fi i; k {x) = 

for all fc = 1, . . . , n — r and all x G Co- I n particular, this means that 

vm* ) • IfiJAix ) = 0, 

that is, [fi, fj](x°) G ker^/»(a; ) = A(cc ), proving involutivity I 

If Oo is an open subset of O, A is a distribution on O, and II : O — * V 
is a diffeomorphism onto some open subset V C 1", we denote by II* A the 
distribution on V defined by attaching to each element of V the vector space 
associated to its preimage: 

(ILA)(z) := II, (U-\z)) ■ A (IT 1 ^)) . (4.43) 

If / G Y(0), we also consider the vector field on V: 

(ILJ)(z) := IMII- 1 ^)) ■ f(H- l (z)) . (4.44) 

One may interpret 11*/ as the vector field obtained when changing coordinates 
under z = II(a;), in the sense that, if x(-) is a curve on O so that x = f(x), 
then z(t) := H(x(t)) satisfies, clearly, the differential equation z = (II sf /)(z). In 
other words, the flow of / maps into the flow of 11*/. Frobenius' Theorem can 
be reinterpreted in terms of such coordinate changes: 

Exercise 4.4.17 Suppose that A = A.f lt „.j r is a distribution of constant rank 
r on an open set O C R". Show that the following two properties are equivalent: 

• A is involutive. 
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• For each x° G O there is an open subset Oq C O which contains x° and 
a diffeomorphism II : Oq — > V into some open subset V C l n such that 
(II* A)(z) = span {ei, . . . , e r } for each z e V, where e, is the ith canonical 
basis vector. 

(A distribution of the form A eii .... Cr is said to be a flat distribution. So, the claim 
is that a distribution is involutive if and only if it is locally flat in appropriate 
coordinates.) □ 

Exercise 4.4.18 Suppose that fi, ■ • ■ , f r are smooth vector fields on an open 
set O C R™, and that fi(x), . . . , f r (%) are linearly independent for each x € O. 
Show that the following two properties are equivalent: 

• [fi, fj] = for each i,j € {1, . . . , r}. 

• For each x° € O there is an open subset Oq C which contains a; and 
a diffeomorphism II : Oq — > V into some open subset V C l n such that 
(Tl«fi)(z) = e; for each z G V, where ej is the ith canonical basis vector. 

(That is, the vector fields commute if and only if there is a local change of 
variables where they all become /; = ej.) □ 

Exercise 4.4.19 Provide an example of a set of vector fields fi, ■ • ■ , f r so that 
(a) the distribution A = A^ x y r has constant rank r and, locally about each 
point x° G 0, transforms under a diffeomorphism into A ei .... er (in the sense 
of Exercise 4.4.17), but (b) there is some point x° such that, for no possible 
diffeomorphism II defined in a neighborhood of x°, (Tl*fi)(z) = e; for all i. □ 



Exercise 4.4.20 Consider the following vector fields in 



p2. 



(a) Check that [/i,/2] = 0. Explain in geometric terms why e tl ^ 1 e t2 f 2 x° = 
e t2f2 e tih x o for all tl) t2) and ^,0^ 

(b) Find, locally about each a; ^ 0, an explicit change of coordinates II (whose 
existence is assured by (a), cf. Exercise 4.4.18) so that /i becomes e\ and fa 
becomes e-i under II. □ 

Exercise 4.4.21 Consider the following partial differential equation: 

(p Xl {x 1 ,x 2 )xj + ip X2 {xi,x 2 ) = 0. 

(a) Explain how we know that, locally about each x° = (cc?,^) £ R 2 > there 
exists a solution ip of this equation so that ip(x°) = and \7ip(x) ^ everywhere 
in its domain. 

(b) Now find one such solution, about each point. (You will need different 
functions tp around different points.) 

(Hint: You may want to introduce an appropriate vector field and follow the 
construction in the proof of Lemma 4.4.16.) □ 
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4.5 Necessity of Accessibility Rank Condition 

We wish to prove now two partial converses of Theorem 9 (p. 156). The first one, 

<T 

valid for arbitrary smooth systems, says that if the sets 1Z^ (x) have nonempty 
interior (for all T, V) for each x in an open and dense subset of X, then the acces- 
sibility rank condition also holds in a (possibly different) open and dense subset 
of X. The second one, valid for analytic systems, drops the "possibly differ- 
ent" . (The counterexamples in Exercise 4.3.6 show that more than smoothness 
is needed in order to prove this stronger statement.) 

Given any system x = f{x,u), the set of points where the accessibility rank 
condition holds is an open set (possibly empty), because if X\(x), . . . , X n (x) are 
linearly independent, then they are also linearly independent in a neighborhood 
of x. Thus, asking that the rank condition holds in a dense set is equivalent to 
asking that it holds in an open dense set. 

Theorem 11 The following statements are equivalent: 

1. There is an open and dense subset Xi C X such that int7\Ly (x ) ^ for 
each x° G Xi, each neighborhood V of x , and each T > 0. 

2. There is an open and dense subset X 2 C X such that the accessibility rank 
condition holds at each x° G X 2 . 

Proof. Theorem 9 (p. 156) shows that if the accessibility rank condition holds 
for each x° G X 2 , then mtTZy (x a ) ^ for each x° G X 2 and all T, V. Assume 
now, conversely, that there is a set Xi as in the first statement. Suppose that 
the second statement is not true. In that case, there is some open subset Xo C X 
such that the accessibility rank condition fails for every x° G Xo- (Otherwise, 
the set where it holds is dense, and hence open dense.) Since Xo is open and Xi 
is open dense, the intersection Xi |"| Xo is a nonempty open set, so we may, and 
will, assume without loss of generality that Xo C Xi. Let r < n be defined as 
the maximal possible dimension of the vector spaces Ai,a{x), over all x G Xo- 

Pick X\, . . . , X r G *4la an d x ° € Xo so that Xi(x°), . . . , X r (x°) are linearly 
independent. Then, Xi (x) , . . . , X r (x) are linearly independent for all x near 
x°. Restricting Xo if needed, we may assume that X\{x), . . . , X r {x) are linearly 
independent for all x G Xo, and they generate Ala(x) (because of maximality 
of r). We introduce the constant-rank distribution A := Ax lt ....x r - We have 
that [Xi, Xj] G -4-la for all i,j, because, by definition, Ala is a Lie algebra of 
vector fields. By the equivalence 3a <£4> 3b in Lemma 4.4.9, A is involutive. 

We apply Lemma 4.4.16 (Frobenius), to conclude that there is some open 
subset V C Xo containing x°, an e > 0, and a diffeomorphism II : V — > (— e, e) n , 
so that, if II 2 : V — > (— e, e) n ~ r denotes the last n — r coordinates of II, then 

(n 2 ),(aO/(z,«) = (4.45) 

for all u G IX and x G V (using the fact that, by definition, f u G A C Ala, so 
fu(x) = f{x,u) G A(x) for all x). 
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We claim that IZy (x°) has an empty interior, in fact for every T > 0, which 
contradicts x° G Xi. Since II is a diffeomorphism, it will be enough to show 
that the image 

n(7^V)) 

has an empty interior. 

Pick any control ui G £^(0,T), admissible for x°, with the property that 
the path x{t) := <f>(t, 0, x° , u>) satisfies x{t) G V for all < t < T. Let z{t) := 
Tl(x(t)). This is absolutely continuous, and 

z(t) = n.(ar(t)) /(»(*),«(«)) = g(z(t)Mt)) 

for almost all t G [0,T], where we define g u := ![*/„ (cf. Equation (4.44)): 

g(z,u) := II. (n- 1 ^)) f(U-\z),u) 

for each z G {—e,e) n and each u G 11. We partition z into blocks of size n — r 
and r respectively, so that the differential equation satisfied by z can be written 
as z\ = g\ (z\ , z<i , u) , £2 = gi{z\, Z2,u). By (4.45), 32 = 0. Thus the equation is 

zi = gi{zi,z 2 ,u) 

Z2 = 0, 

in the z variables, and this shows that 

u(nl T (x )) C (-£,£)»■ xn 2 (a; ) 
has empty interior. I 

Analytic Case 

We now consider analytic systems x = f(x,u), meaning that f u = /(-,«) is 
analytic, for each u G U. Models derived from classical mechanics are most 
often analytic. 

Lemma 4.5.1 Suppose that the accessibility rank condition holds at z° = 
e t ^ u x°, for some t, x°, and u G 11, and the system is analytic. Then the acces- 
sibility rank condition holds at x° as well. 

Proof. Let X = f u . Pick a set Y\, . . . , Y n of vector fields in *4la such that 

Y 1 (e tx x°),...,Y n (e tx x°) 

are linearly independent. By Corollary 4.4.6, the vectors a,d x Yj(x°) span 1™. 
The vector fields ad^Yj are in ^IlAi because X as well as the Yj's are. Therefore 
-4la(^°) = K ra , that is, the accessibility rank condition holds at x . I 
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Theorem 12 For analytic systems, the following two properties are equivalent: 

• For each state x and neighborhood V of x, and each T > 0, int IZy (x) ^ 0. 

• The accessibility rank condition holds at every state x. 

Proof. Theorem 9 (p. 156) established the sufficiency of the rank condition. 
Suppose now that int7\Ly (x) ^ for all x, V,T. By Theorem 11 (p. 177), the 
accessibility rank condition holds at every state z in some open and dense subset 
Z of X. Pick any x a £ X. Consider the set 

W := mtU{x°)f]Z. 

This set is open (intersection of open sets) and nonempty (because int1Z(x°) is 
open and Z is dense). Pick any point z° £ W; thus, there is some T > and 
some u) £ C^f(0, T) such that z° = 4>{T, 0, x°, lu). Approximating u) by piecewise 
constant controls, we can conclude that there is a convergent sequence of states 
Zk — > Z , each Zk reachable from x° by piecewise constant controls (see Lemma 
2.8.2). Pick any k large enough so that Zk € Z. Then, the accessibility rank 
condition holds at Zk, and there exist U\,...,ui € IX and positive t\, . . . , ti such 
that e* 1 -^"! . . .e*""«x° = Zk- Applying Lemma 4.5.1 t times, we have that the 
accessibility rank condition holds at x°. I 
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Exercise 4.6.1 Let G : X — > R nxm be a matrix function of class C°°, defined 
on an open subset X C R™. Consider the following properties: 

(C) For each x° G X, there is some neighborhood X of x°, and there are two 
matrix functions C : X -> R nxn and D : X -^ R mxm , both of class C°° 
and nonsingular for each a; € Xo, such that 

C(x)G(a;)D(aj) = (g) for all x £ X , 

where / is the m x m identity matrix. 

(F) For each a; G X there are Xo, C, and Z) as above, and there is some 
diffeomorphism II : Xo — > Vo into an open subset of R™ so that C(a;) = 
n„(cc) for all i£ lo. 

(a) Show that (C) holds if and only if rank G(x) = m for all x £ X. 

(b) Show that (F) holds if and only if the columns gi,. . . , g m generate a dis- 
tribution A gi! ....g m which is involutive and has constant rank m. (That is, 
rankG(a;) = m for all x £ X and the n 2 Lie brackets [<?i, <?j] are pointwise linear 
combinations of the gi's.) 
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(c) Give an example where (C) holds but (F) fails. 

(d) Interpret (b), for a system without drift x = G(x)u, in terms of a change 
of variables (a feedback equivalence) (x,u) *— » (z,v) := (I[(x), D(x)^ 1 u). What 
are the equations for z in terms of v, seen as a new input? 

(Hint: (a) and (b) are both easy, from results already given.) □ 

Exercise 4.6.2 Suppose that A = Ax lt ...,x T has constant rank r, is involu- 
tive, and is invariant under the vector field /. Pick any x° € O, and let Oq 
and II be as in Frobenius' Lemma 4.4.16. Define, for z £ (—£,£)", g(z) := 
n*(II (z)) /(II (2)), and partition g = (51,92)' an d Z = (zi,z 2 )' as in the 
proof of Theorem 11. Show that g 2 does not depend on z\, that is to say, the 
differential equation x = fix) transforms in the new coordinates z = H(x) into: 

z\ = gi(zi,z 2 ) 

Z2 = gi(zi)- 

Explain how, for linear systems x = Ax, this relates to the following fact 
from linear algebra: if A has an invariant subspace, then there is a change of 
coordinates so that A is brought into upper triangular form consistent with 
that subspace. (Hint: (For the proof that dg 2 /dz\ = 0.) We have that 
52(11(2;)) = Tl2*(x)f(x). On the other hand, each row of Tl 2 *(x)f(x) is of the 
form Lfipi, where tp^s are the rows of n 2 . We know that Lx-tpi = f° r a U h J 
(this is what Lemma 4.4.16 gives), and also Lif x-V^i = (because A is invari- 
ant under /), so conclude that Lx(Lftpi) = 0. This gives that the directional 
derivatives of the rows of g2*(n(x)) along the directions ej(x) := Tl*(x)Xj(x) 
are all zero. Now observe that the vectors ei(x) are all of the form (en, 0)', and 
they are linearly independent.) □ 

Exercise 4.6.3 Use the conclusion of Exercise 4.6.2 to prove: for a control- 
affine system x = go+^2 Ui9i, if A = Ax L ,....x r has constant rank r, is involutive, 
and is invariant under go, and in addition all the vector fields gi G p A for 
i = 1, . . . , m, then there is a local change of coordinates, about each state, in 
which the system can be expressed in the form 

m 

zi = 9l{z l ,z 2 ) J r^ j u l g}(z 1 ,z 2 ) 

8 = 1 

z-2 = gl(z 2 ) . 

Compare with the Kalman controllability decomposition for linear systems. □ 

Exercise 4.6.4 Show that, for analytic systems with connected state space X, 
if the accessibility rank condition holds at even one point a; G X, then it must 
also hold on an open dense subset of X. (Hint: Use that if an analytic function 
vanishes in some open set, then it must be identically zero.) □ 
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4.7 Notes and Comments 

The systematic study of controllability of nonlinear continuous-time systems 
started in the early 1970s. The material in this chapter is based on the early work 
in the papers [255], [284], [391], and [185], which in turn built upon previous 
PDE work in [91] and [182]. Current textbook references are [199] and [311]. 
An excellent exposition of Frobenius' Theorem, as well as many other classical 
facts about Lie analysis, can be found in [53]. 

Under extra conditions such as reversibility, we obtained an equivalence be- 
tween controllability and accessibility. Another set of conditions that allows 
this equivalence is based on having suitable Hamiltonian structures, see, e.g., 
[51]. The advantage of considering accessibility instead of controllability is that, 
in effect, one is dealing with the transitivity of a group (as opposed to only a 
semigroup) action, since positive times do not play a distinguishing role in the 
accessibility rank condition. (Indeed, for analytic systems on connected state 
spaces, accessibility is exactly the same as controllability using possibly "neg- 
ative time" motions in which the differential equation is solved backward in 
time.) 

Also, for continuous-time systems evolving on Lie groups according to right- 
invariant vector fields, better results on controllability can be obtained; see, for 
instance, [126], [211], and [351]. Nonlinear global controllability problems are 
emphasized in [160], [161], and [162]. 

The question of providing necessary and sufficient characterizations for con- 
trollability is still open even for relatively simple classes such as bilinear contin- 
uous-time systems, but substantial and very deep work has been done to find 
sufficient conditions for controllability; see, for instance, [388], or [239]. It is 
worth remarking that it can be proved formally that accessibility is "easier" 
to check than controllability, in the computer-science sense of computational 
complexity and NP-hard problems; see [366], [240]. 

For discrete-time nonlinear accessibility, see for instance [206] and [208]. 

Example 4.3.13 is from [308], which introduced the model and pursued the 
analysis of controllability in Lie-algebraic terms. 

Much recent work in nonlinear control has dealt with the formulation of 
explicit algorithms for finding controls that steer a given state to a desired 
target state, under the assumption that the system is controllable. This is 
known as the path-planning problem. See for instance the survey paper [251] 
and the many articles in the edited book [30] . 
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Chapter 5 

Feedback and Stabilization 



The introductory Sections 1.2 to 1.5, which the reader is advised to review at 
this point, motivated the search for feedback laws to control systems. One is 
led then to the general study of the effect of feedback and more generally to 
questions of stability for linear and nonlinear systems. This Chapter develops 
basic facts about linear feedback and related topics in the algebraic theory of 
control systems including a proof of the Pole-Shifting Theorem described in 
Chapter 1, as well as an elementary introduction to Lyapunov's direct method 
and a proof of a "linearization principle" for stability. Some more "advanced" 
topics on nonlinear stabilization are also included, mostly to indicate some of 
the directions of current research. 

5.1 Constant Linear Feedback 

In this Section, K is an arbitrary field; unless otherwise stated, (A, B) denotes 
an arbitrary but fixed pair with A € K" x ™ and B € K™ xm , and a "continuous- 
time" or a "discrete-time" system means a linear and time-invariant system. In 
the special case m = 1 , we often denote B as lowercase b. 

It is natural to study systems under a change of basis in the state space, 
X = Tz. We already considered such transformations when dealing with the 
Kalman controllability decomposition (Lemma 3.3.3). Recall that S n ^ m is the 
set of all pairs with given n, m, and S^ m is the set of all such controllable pairs. 

Definition 5.1.1 Let (A, B) and (A,B) be two pairs in S n ^ m . Then (A, B) is 
similar to (A, B), denoted 

(A,B)~(A,B) 



T~ X AT = A and T~ X B = B 



if 

for some T G GL(n). D 
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Lemma/Exercise 5.1.2 The following facts hold for the relation ~: 

1. It is an equivalence relation on S nim . 

2. If (A, B) ~ {A, B), {A, B) is controllable if and only if (A, B) is. 
Lemma 5.1.3 Let (A, b) be any pair with m = 1, and write 

X(s) = det(s/ - A) = s n - a n s n ~ x - ... - a 2 s - a 1 
for the characteristic polynomial of A. With the matrices 

ai \ 

a 2 
a 3 



□ 



^ := 



/0 

1 
1 



Vo 



6+ := 









1 



w 



Vo/ 



it follows that 



AR(4,6) = R(A,6)A f and b = R(A,b)tf , 



where R = R(A, 6) is the controllability matrix of (A, b). 

In particular, (A, b) is controllable if and only if it is similar to (A' , b'). 

Proof. The first part is an immediate consequence of the Cayley-Hamilton 
Theorem. Since (A^b^) is controllable, because 

R(A t ,6 t ) =/, 

the sufficiency in the last statement follows from Lemma 5.1.2; for the necessity 
part, T := R(A, b) provides the required similarity. I 

The system (A',b*) is sometimes called the controllability form of (A, b). 
From the above Lemma and the fact that (A' , b') depends only on the charac- 
teristic polynomial of A, and because ~ is an equivalence relation, we conclude 
the following. It says that, for single-input controllable systems (m = 1), the 
characteristic polynomial of A uniquely determines the pair (A, b) up to simi- 
larity. 

Corollary 5.1.4 Assume that (A,b) and (A, b) are two pairs in S^ ll . Then, 
(A, b) is similar to (A, b) if and only if the characteristic polynomials of A and 
A are the same. □ 

Definition 5.1.5 The controller form associated to the pair (A,b) is the pair 



(CF) 



A h := 


/ 1 ... \ 
1 ... 

... 1 

\ai a 2 a 3 ... a n ) 


& := 





W 



where s n — a n s r ' 



a 2 s — ct\ is the characteristic polynomial of A. 



□ 
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The symbols A b and b b are read as "A-flat" and "b-flat," respectively, the 
terminology motivated by the fact that the nontrivial coefficients appear hori- 
zontally along a row, in contrast to the controllability form, where they appear 
vertically. 

Lemma/Exercise 5.1.6 The pair (A , b ) is controllable, and the characteris- 
tic polynomial of A is 

n n—1 r—i 

S — a n s — ... — Q^S — Ql . U 

Proposition 5.1.7 The single-input (m = 1) pair (A, b) is controllable if and 
only if it is similar to its controller form. 

Proof. Assume that the pair is controllable. Then, since A has the same 
characteristic polynomial as A, and the pair (A , b ) is controllable, it follows 
from Corollary 5.1.4 that (A, b) is similar to its controller form. Conversely, if 
this similarity holds, then the second statement in Lemma 5.1.2 says that (^4, b) 
is controllable, because (A , b ) is controllable. I 

Thus, up to a change of variables, single-input controllable linear, time-in- 
variant systems are precisely those obtained from nth order constant coefficient 
scalar differential equations 

x^ - a n x (n ~^ - ... - axx = u 

by the usual introduction of variables Xi := x^ 1-1 ' , and analogously for discrete- 
time systems. Recall that in Chapter 1 we saw how the notion of state-space 
system was motivated by this construction. 

We shall be interested in studying the effect of constant linear feedback. The 
following Lemma will be helpful in that regard: 

Lemma/Exercise 5.1.8 For any pair (A, B) and any F € K mxn , 

K{A + BF,B) =K{A,B). 

In particular, (A + BF, B) is controllable if and only if (A, B) is. □ 

Definition 5.1.9 The nth degree monic polynomial \ * s assignable for the 
pair {A, B) if there exists a matrix F such that x A , RP = X- ^ 

Lemma 5.1.10 If (A,B) ~ (A, B), then they can be assigned the same poly- 
nomials. 

Proof. Given any F, it holds that 

^A+BF ~ ^A+BF 

provided that one chooses F := FT. I 
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In Chapter 8 it will be proved using optimal control techniques (Corollary 
8.4.3 and Exercise 8.4.5) that, if IK = R and (A, B) is controllable, then there 
is a feedback matrix F such that A + BF is Hurwitz and another F such that 
A+ BF is convergent (discrete-time Hurwitz). As discussed in the introductory 
Section 1.5, in fact one can assign an arbitrary spectrum under feedback, as 
was illustrated there with the linearized pendulum. This is the Pole-Shifting 
Theorem, to be proved below. The use of the term pole to refer to the eigenvalues 
of A originates in the classical frequency-domain theory of linear systems. 

Recall (Definition 3.3.5) that \ is the uncontrollable part of the character- 
istic polynomial of A, and we let \ = 1 if the pair (A, B) is controllable. 

Theorem 13 (Pole-Shifting Theorem) For each pair (A,B), the assignable 
polynomials are precisely those of the form 



*A+BF y~\X-u 



where \ is an arbitrary monic polynomial of degree r = dim 1Z(A, B). 

In particular, the pair (A, B) is controllable if and only if every nth degree 
monic polynomial can be assigned to it. 

Proof. Assume first that {A, B) is not controllable. By Lemmas 3.3.3 and 
5.1.10, we can assume without loss of generality that (A, B) has the form that 
(A, B) has in (3.6). Write any F € ]g mx ™ i n t ne partitioned form {F\,F-2), 
where F\ is of size m x r and Fi is of size m x (n — r). Since 

*A+BF — X Al+BlFl X u i 

it follows that any assignable polynomial has the factored form in the statement 
of the Theorem. 

Conversely, we want to show that the first factor can be made arbitrary by 
a suitable choice of F (except for the case r = 0, where only \ is there and 
hence there is nothing to prove). Assume given any v . If we find an F\ so that 

^A 1 +B 1 F 1 ~ *i ' 

then the choice F := (i<\,0) will provide the desired characteristic polynomial. 
By Lemma 3.3.4, the pair (Ai,B\) is controllable. Thus, we are left to prove 
only that controllable systems can be arbitrarily assigned, so we take (A, B) 
from now on to be controllable. 

When m = 1, we may assume, because of Lemmas 5.1.10 and 5.1.7, that 
(A, b) is in controller form. Given any polynomial 

X = s n - P^- 1 - . . . - p 2 s - 0! 

to be assigned, the (unique) choice 

/ := (ft - ai,. ..,/?„ - a n ) 
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satisfies x = %. Thus, the result has been proved in the case m = 1. (The 

fact that / can be obtained immediately from the controller form justifies the 
terminology "controller form." ) 

Now let m be arbitrary. Pick any vector »6ll = K m such that Bv ^ 0, and 
let b := Bv. We next show that there exists an F\ G ]g mx " such that 

(A + BF lt b) 

is itself controllable. Because the result has been established for the case when 
m = 1, there will then be, for any desired \i a (unique) 1 x n matrix / such 
that the characteristic polynomial of 

A + B{F X + vf) = {A + BFi) + bf 

is x, and the Theorem will be proved using the feedback F := F\ + vf. 
To establish the existence of F\, we argue as follows. Let 

{xi,... ,x k } 

be any sequence of linearly independent elements of X, with k as large as pos- 
sible, having the properties that X\ = Bv and 

Xi - Ax^i e B := col B (5.1) 

for i = 1, . . . , k (with xq := 0). In other words, the sequence is required to be 
such that 

Bv = x x yx 2 ^ ...yx k 

and k is as large as possible. 

We claim that k = n. Consider the span V of {x\, . . . ,x k }. By maximality 
of k, 

Ax k + BueV for all ueU. (5.2) 

This implies, in particular, that 

Ax k e V . 

Thus, equation (5.2) shows that 

BCV-{Ax k }CV, 

so equation (5.1) implies that 

Axi e V, l = l,...,k-l. 

We conclude that V is an ^-invariant subspace that contains B. It follows that 
1Z(A, B) C V. By controllability, this implies that indeed k = n. 
Finally, for each i = 1, . . . , k — 1, we let 

r \Xi . — Ui , 
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where u, £ 11 is any element such that 

Xi - Axi-i = Bui-i , 
and we define F\Xk arbitrarily. Since 

K(A + BF 1 ,x 1 ) = (x 1 ,...,x n ), 
the desired single-input controllability is achieved. I 

Exercise 5.1.11 O Show that, if the field IK is infinite, (A,B) is controllable, 
and the matrix A is cyclic, then there exists some u so that (A, Bu) is control- 
lable. Give counterexamples to show that this fails if the hypothesis that A is 
cyclic is dropped, and to show that even if A is cyclic the result fails over the 
field of two elements. (Hint: Use the fact that the geometric multiplicity of each 
eigenvalue of the transpose A' must be one, and then solve the inequalities 

v'Bu ^ 

for each eigenvector v of A' .) □ 

Exercise 5.1.12 O For single-input systems, it is possible to give explicit for- 
mulas for the (unique) feedback law that assigns any given polynomial. One of 
these, Ackermann's formula, is as follows: 

/ = -(0---01)R(A,b)- 1 X (A), 

where \ ls the desired polynomial. Prove this formula. □ 

Remark 5.1.13 A quick numerical procedure for pole-shifting when K = K 
(or C) is based on the following argument. Given a controllable pair (A, B), 
we first pick an arbitrary nonzero column b of B and then generate at random 
(using a pseudorandom number generator) a matrix F. With probability one, 

{A + BF, b) 

is controllable. Indeed, the set of F for which the property is true is the com- 
plement of an algebraic set, hence open dense and of full measure if nonempty; 
as we know from the proof of the Theorem that there is some such F, this set is 
open dense. (If the random choice happens to be one of the exceptional cases, 
we just generate another F .) Now Ackermann's formula (Exercise 5.1.12) can 
be applied to (A + BF, b). □ 

Exercise 5.1.14 Refer to Exercise 3.2.12. 

(a) Find the controller form for E and the similarity T needed to bring the 
original system into this form. (In finding the latter, note that T : (A, b) ~ 
(A b ,6 b )isgivenbyT = R(R b )" 1 , where R = R(A,b) and R b = R(A b ,6 b ). 
Take all constants equal to one, for simplicity.) 

(b) Find a feedback u = fx so that all eigenvalues of A + bf equal — 1. □ 

Exercise 5.1.15 Repeat Exercise 5.1.14 for the system E in Exercise 3.2.13, 
taking for simplicity M = m = F = g = l=l. □ 
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5.2 Feedback Equivalence* 

The controller form (A , b ) in Definition 5.1.5 was useful in establishing the 
Pole-Shifting Theorem. We proved that every controllable single-input system 
can be brought into (a unique) controller form under change of basis in the 
state space. Further, a system in this form then can be transformed easily 
under feedback into any other. In particular, from the form (CF) and using the 
transformation 

(>,& b )-> (^ b -6 b (oi,a2,a 3) ...,a n ),6 b ) 



results the followin 


g pair {A n ,b n 




/0 1 .. 
'001.. 


JT-Tl = 






0.. 
\0 .. 







1 

0/ 



b„ 









Vi/ 



(5.3) 



The above transformation can be interpreted as the application of the feedback 

u= fx = -(c*i, a 2 , a 3 , . . . , a n )x , 

and essentially the previous results show that, under change of basis and feed- 
back, any single- input continuous-time time-invariant linear controllable system 
can be reduced to a scalar nth order equation of the form 



» 



u, 



which corresponds to an nth order integrator (or an n-step delay, in the case of 
discrete-time systems); see Figure 5.1. 



l n-l 



Figure 5.1: Cascade of integrators. 

The purpose of this Section is to provide a generalization of this fact to the 
case of multi- input (m > 1) systems. It will state that every controllable pair 
(A, B) can be made equivalent, in a suitable sense to be defined below, to one 
in a form analogous to that obtained for the case m = 1; for continuous-time 
systems, this will result in the form 



l i 

>2) 



«1 

«2 



,.("»•) 



* This section can be skipped with no loss of continuity. 
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where (ki, . . . , n r ), r = rank B, is a sequence of integers, called the controllability 
indices or Kronecker indices of the system. These integers satisfy 

Ki + . . . + n r = n 

and are unique up to ordering. Thus, every linear continuous-time time-invari- 
ant system can be made to behave like r parallel cascades of integrators, under 
the action of feedback. This result is of some theoretical interest and can be used 
to give an alternative proof of the Pole-Shifting Theorem. In addition, because 
the Kj's are unique, the result provides an effective test to determine whether 
two systems are equivalent under feedback transformations, in the sense defined 
below. 

Except when explaining the interpretations for continuous-time systems, we 
assume that K is an arbitrary field. 

Definition 5.2.1 Let (A, B) and (A, B) be two pairs in S n ^ m . Then (A, B) is 
feedback equivalent to (A,B), denoted 

(A,B) = (A,B) 

»/ 

T~ 1 (A + BF)T = A and T~ 1 BV = B 

for some T € GL{n), some F € K mxn , and some V € GL(m). □ 

Feedback equivalence corresponds to changing basis in the state and control- 
value spaces (invertible matrices T and V, respectively) and applying a feedback 
transformation u = Fx + v! ', where u' is a new control. 

Lemma/Exercise 5.2.2 The following facts hold for the relation =: 

1. It is an equivalence relation on iS„ ;m . 

2. If (A, B) ~ (2, B), then (A, B) = {A, B). 

3. If (A, B) = (A, B), then (A, B) is controllable iff (A, B) is. □ 

The notion of feedback equivalence is closely related with Kronecker 's work 
(ca. 1890) on invariants of matrix pencils. This connection is explained in the 
next problem; we will derive our results directly in terms of feedback equivalence, 
however. 

Exercise 5.2.3 An expression of the type sF + G, where F and G are two 
n x I matrices over K, is called a pencil of matrices. We think of this as a formal 
polynomial in s, with equality of sF+G and sF' + G" meaning pairwise equality: 
F = F' and G = G' . Two pencils are called equivalent if there exist invertible 
matrices T € GL(n) and S G GUI) such that 

T[sF + G}S= [sF' + G 1 ]. 
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Given a pair (A, B), consider the pencil 

[ 8 I-A,B] = s[I,0] + [-A,B\, 

with I = n + m. Prove: (A, B) = (A, B) if and only if the corresponding pencils 
are equivalent. □ 

A partition k = («i, . . . , K r ) of the integer n is a sequence of positive integers 

K\ > K 2 > ■ ■ ■ > K r 

such that 

Kl + K,2 + ■ ■ ■ + K r = H ■ 

Associated to each such partition for which r < m, we consider the pair (A K , B K ), 
defined (with the same n,m) as follows. The matrix A K is partitioned into r 2 
blocks, and B K is partitioned into mr blocks: 



(A Kl ••• \ /b Kl ••• ••• 0\ 

/ a ... ' " ' » " 



6 K2 ••• ••• 



••• \ /6 K , ••• ••• 

B 

Vo ■■■ A K J Vo ■•• b K ■■■ oJ 

(5.4) 
where each A Ki has size Kj x m and is as in equation (5.3), and each b Ki is a 
column of length Ki as in that equation. Note that when m = 1 there is only 
one partition, k = (n), and (A K ,B K ) is in that case just the pair (A n ,b n ) in 
(5.3). The above pair (A K ,B K ) is said to be in Brunovsky form. For each k, 
this is a controllable pair. Note that 

rank B K = r < m 

and that a continuous-time system whose matrices have this form consists of r 
integrator cascades, as discussed informally above. 

The main Theorem to be proved in this section is as follows: 

Theorem 14 For every pair (A, B) in S„ m , there is a unique partition n such 
that{A,B) = {A Kl B K ). 

This implies that the partition n is a complete invariant for controllable 
systems under feedback equivalence and that the number of equivalence classes 
is equal to the number of such partitions for which r < m. For instance, if n = 3 
and m = 2, then there are exactly two equivalence classes, corresponding to the 
partitions (3) and (2,1). 

The proof of Theorem 14 will be organized as follows. First we show how 
a partition k can be associated to each pair. We show that this assignment is 
invariant under feedback equivalence, that is, that two feedback equivalent pairs 
necessarily must give rise to the same k, and further, for the system (A K ,B K ) 
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already in Brunovsky form (5.4) the associated partition is n. This will prove 
the uniqueness part of the statement. To establish existence, we show that, for 
the k associated to the pair (A, B), it holds that (A, B) = (A K , B K ); this is done 
by an explicit construction of the needed transformations. 

Fix any controllable pair (A,B). Let bi denote the ith column of B, and 
consider the sequence of vectors 



h,b 2 



,Ab 1 ,Ab 2 , 



,Ab rt 



,A n - 1 bi,A' 



,A n - 1 6„ 



We say that a vector in this sequence is dependent if it is a linear combination 
of the previous ones. Observe that if some A J bi is dependent, and j < n — 1, 
then A^ +1 bi also must be dependent. If a vector is not dependent, we call it 
independent. We now arrange these vectors in the following pattern: 

b 1 b 2 

Ab\ Ab 2 ■ ■ ■ Ab. n 



A n - X b x A r 



(5.5) 



A n- 



and we associate a preliminary partition A = (Ai, . . . , A s ) of n as follows. The 
integer A^ is defined as the number of independent vectors (in the sense explained 
above) in the fth row of the table (5.5), and s is the largest index so that the 
sth row contains at least one independent vector. From the remark about 
the dependence of A^ +l bi following from that of A J bi, we know that this is a 
nonincreasing sequence. Further, 

Ai = rank_B < m , 

and by controllability 

Ai + ... + A s = n, 

so that this is a partition of n into parts none of which exceeds m. If we need to 
emphasize that this was done for the system (A,B), we write X(A, B) instead 
of just A. 





s 










X 


2 

\ 
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f 




s 


1 ^ 




V 


) 







Figure 5.2: Example leading to indices (3,1). 

As an illustration, take the following system, with n = A and m = 2 (see 
Figure 5.2): 

1 0\ /0 1\ 

.0 10/ \0 0/ 
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for which, denoting by e, the ith canonical basis element in K 4 , the above 
pattern is 

e-i ei 

ei e 3 

e 3 e 4 
e 4 

and A(vl,-B) = (2,1,1). 

Lemma/Exercise 5.2.4 If (A,B) = (A,B), then X(A,B) = X{A,B). {Hint: 
Establish that for each k the rank of 

[B,AB,...,A k B] 

remains invariant under each of the possible transformations B t— » BV , A t— > 

rU^ii-vA + BF.) D 

The controllability indices of (A, £>) will be obtained from A(A, B) as the 
"conjugate" partition to A, in the following sense: 

Definition 5.2.5 Given a partition A = (Ai, . . . , A s ) of n, the conjugate par- 
tition to A (also called the "dual" partition to X) is the partition A' given by 
(Ai, . . . , A' s ,), where \[ is the number of parts of X that are > i. □ 

For instance, the dual partition to (2, 1, 1) is (3, 1), since there are three A»'s 
larger than or equal to 1, and just one larger than or equal to 2 (and none larger 
than 2). 

The following concept from combinatorics is useful when studying partitions. 
A Young tableaux is a matrix of zeros and ones, with the property that there 
is never a one below nor to the right of any zero, and no row or column is 
identically zero. 

Given any partition A = (Ai,...,A s ), we can associate to it the Young 
tableaux of size s x Ai that has Ai ones in the first row, A 2 in the second, and 
so forth. For instance, associated to (2, 1, 1) is the tableaux 

1 1 
1 
1 

Conversely, each tableaux with n ones comes from precisely one partition of n. 
Under this rule, the tableaux of the conjugate is the transpose of the original 
tableaux. From this, it follows that A" = A for all A, so A is uniquely determined 
by A'. Note also that, always, s' = Ai. 

Definition 5.2.6 The controllability indices k(A, B) of the controllable pair 
(A, B) are the elements of the sequence m, . ■ . , K r , where 

k = (Kl, . . . , K r ) 

is the partition conjugate to X(A,B). D 



194 5. Feedback and Stabilization 

For example, the system (5.6) has indices 3, 1. In general, the indices are 
the numbers of independent vectors in the columns of (5.5), arranged in nonin- 
creasing order. 

From Lemma 5.2.4 it follows that controllability indices are invariant under 
feedback equivalence. Observe that r = Xi < m, the rank of B. The assignment 
(A, B) —> k(A, B) is a mapping from <S^ m into the set of all partitions of n into 
r = rank B parts. 

Lemma/Exercise 5.2.7 For each partition k of n into r parts, 

«(i4 R , B*) = k . □ 

The uniqueness statement in Theorem 14 now follows: If (A, B) = (A K >, B K >) 
and also (A, B) = (A K », B K >>), then because of feedback equivalence and Lemma 
5.2.7, 

k' = k(A h , ,B k ,) = k(A, B) = K(A K n , B K n ) = K " , 

and the Brunovsky form is unique. 

Next we show that there is a sequence of transformations as in the defini- 
tion of feedback equivalence that brings any given pair into the corresponding 
Brunovsky form. 

We first apply a transformation consisting of a permutation of the columns 
of B (i.e., V is a permutation matrix, and T = I, F = 0) in such a way that 
the columns in (5.5) are rearranged so that the number of independent vectors 
in the ith column is nonincreasing. (In the above example, this means that we 
use 

1 



y -'i 



and obtain the new arrangement 

ei e 2 

e 3 ei 

e 4 e 3 

e 4 

after permuting columns.) We denote the new matrix BV again by B and its 
columns by bi. (For the above example this means that now B = [e\,e2\-) 

In terms of the pattern (5.5), then, m is now the number of independent 
vectors in the ith column, and the vectors 

b u ■ ■ ■ , A K i~ 1 b 1 ,b 2 , . . . , A^-%, . . . , b r , . . . , A K "-% 

form a basis of K™. Furthermore, from the definition of dependency it follows 
that for each i there is a linear combination 

r Ki 

A Ki bi + Y, a ij0 A Ki bj = ~YjYj Ui3kA Ki - k bj . (5.7) 

j<i j=l fe=l 
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Next we apply yet another invertible transformation B := BV resulting in new 
columns 

h := hi + ^ aijobj , i = l,...,r, 

3<i 

and bi := bi for i > r, so that, after again calling B just B, we have that we can 
write (5.7) simply as: 

r Hi 

A"*bi + Y.I2 oc ijk A Ki -% = (5.8) 

3 = 1 k=l 

(with different coefficients oiijk's). For each 

i = 1 , . . . , r and I := 1, ...,«» , 
we introduce the vectors 

r l-l 

e u := A'-^ + Y, Yj oiiikA l - 1 - k b j 

j=l fc=l 

(with the convention that eji = 6j). Listing them in the order 
en, e2i, . . . ; e r i, ei2, e22, ...;... 

makes it evident that they are linearly independent, since they form a triangular 
linear combination of the basis vectors 

b 1 ,b 2l ...,b r ,Abi,Ab 2 ,. .. , 

so the matrix T with columns 

[eiie 1 ,ei( (4l _i), . . . ,eii,e 2K2 ,e 2 ( K2 _i), . . . ,e 2 i, . . . , e rKr , e r ( Kr _i), . . . , e r i] 

is invertible. Note that 

r 

Ae^ = e i( ; +1 ) - y^ ajjibj 



for 2 < Kj and that, because of (5.8), 



from which it follows that the matrix on the basis given by the columns of T, 
that is, A := T~ l AT , has the following block structure: 



/A n A 12 
A 2 \ A 22 



A lr \ 

A 2r ' 



(5.9) 



\ J\.f \ -ti-r2 ' ' ' Ayy / 
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where the off-diagonal blocks Aij, i =£ j, have the form 



and the diagonal blocks have the form 



/0 . 

0. 


• °\ 

. 


0. 

\ * * * . 


. 

. */ 


the form 




/0 1 . 

1. 


• °\ 

. 


0. 

\ * * * . 


. 1 

. J 



where the asterisks indicate possibly nonzero entries. Similarly, since en = b{ 
for each i, the matrix T~ l B is in the desired form (5.4) except for the fact that 
the last m — r columns may be nonzero. Applying a new change of basis V in 
the control- value space, these columns can be made zero (since they are linearly 
dependent on the first r rows, because ranki? = r). 

Remark 5.2.8 At this stage, one has not yet applied any purely feedback trans- 
formations (using F /: in the definition of feedback equivalence) but only 
changes of basis on states and controls. The system has been reduced to a form 
analogous to the controllability form (CF). It should be noted however that, 
in contrast to the situation with m = 1, the free parameters left (the aster- 
isks above) are not uniquely determined by the original pair, so this is not a 
"canonical form" in the sense of invariant theory. □ 

Finally, the asterisks in the blocks in (5.9) all can be zeroed by application 
of a feedback transformation A — » A + BF; in fact, from the form of B, this can 
be achieved simply by using as F the negative of the matrix that has those same 
asterisks in the same order (and arbitrary last in — r rows). This completes the 
proof of Theorem 14. I 



1* 



Figure 5.3: Canonical form for example. 



Exercise 5.2.9 Write the matrix [si — A, B] for a pair in Brunovsky form. 
Show how to obtain from this expression, and from Exercise 5.2.3, the fact that 
rank [si — A, B] = n for all s is necessary for controllability. □ 
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Exercise 5.2.10 The Pole-Shifting Theorem can be proved as an easy conse- 
quence of Theorem 14. The critical step is to show that for each controllable 
(A, B) there is some feedback transformation F and some u € K m such that 
(A+BF, Bu) is controllable. Show how to obtain such an F and u from Theorem 

14. □ 

Exercise 5.2.11 Let K = 1 or C. For any fixed n, m, write 

n = sm + t , < t < in 
with s the integer part of n/m. Show that the set of pairs (A,B) with indices 

s + 1, . . 

forms an open dense subset of S„ m . □ 

5.3 Feedback Linearization* 

We study single-input control- affine (time- in variant, continuous-time) systems 

x = f(x) + ug(x). (5.10) 

The state-space is, as usual, an open subset X C W 1 , and the control-value space 
is 11 = K. (This is as in (4.18), with m = 1, but here we prefer to write / and 
g, instead of go and g\ respectively, for the drift and control vector fields.) 

Definition 5.3.1 Let E and E be two systems of the above form, and suppose 
that O and O are open subsets of the respective state spaces X and X. We say 
that (£, O) is feedback equivalent to (E, O) if there exist: 

• a diffeomorphism T : O —> O, and 

• smooth maps a, (3 : O — » M., with f3(x) ^ for all x G O , 
such that, for each x G O: 

r.(aO {fix) + a(x)g(x)) = f(T(x)) (5.11) 

and 

P(x)T*(x)g(x) = g(T(x)) (5.12) 

(f, g and f,g are the vector fields associated to the respective systems). □ 



* This section can be skipped with no loss of continuity. 



198 



5. Feedback and Stabilization 



Notice that X and X must have the same dimension, because T is a dif- 
feomorphism. One can express the two equalities (5.11)-(5.12) in the following 
equivalent form: 



n(x)(f(x)+ug(x)) = f(T(x))- 



1 



P(x) 



u-a(x))g(T(x)) VieO.VuG 



The change of variables 



(x,u) f—» (z,v) := I T{x), 



f3(x) 



[u — a(x)] 



provides a diffeomorphism between Oxl and W X R, whose inverse is 
(z,v) -» (x,u):=(T~ 1 (z),a(T- 1 (z))+p(T- 1 (z))v) . 

Solutions of x = f(x) + ug(x) are transformed into solutions of z = f(z)+vg(z), 
corresponding to the input v = -grp- [u — a(x)}. We may view u = k(x,v) = 
a(x) + f3{x)v as a feedback law which "closes the loop" about the system, with 
v having the role of a new input, to be used for further control purposes, see 
Figure 5.4. Under the change of variables z = T(x), the closed-loop system 





r 




u 








k(x, v) 




x = f{x)+ug(x) 






x 






, 


. 























Figure 5.4: Feedback. 



x = f(x) + k(x,v) g(x) 



(5.13) 



(represented by the dashed box in Figure 5.4) transforms into z = f(z) + vg(z). 
This new system might be easier to control than the original system, especially 
if it is linear. 

Definition 5.3.2 The system £ in (5.10) is feedback linearizable about x°, 

where x° G X, if there exists an open subset OCX containing x° , a controllable 
n-dimensional single-input linear system (£) z = Az + vb, and an open subset 
O C W 1 , such that (S,C) is feedback equivalent to (S,C). □ 

Given any T,a,f3 providing an equivalence as in this definition, T is said 
to be a linearizing change of variables and the map (x, v) t— > a(x) + f3(x)v a 
linearizing feedback. 
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Suppose, as an illustration, that we wish to stabilize about x = x°, and 
assume that we have found a feedback linearization with the extra property 
T(x°) = 0. Since the pair {A,b) is controllable, there is some F G K lxn such 
that z = (A + bF)z is globally asymptotically stable (Pole Shifting Theorem 13 
(p. 186)). Changing back to x variables, we consider the system without inputs 



x = f{x) + k(x)g(x). 



(5.14) 



where the new feedback k{x) := a(x) + f3(x)FT(x) is obtained by substituting 
u = k(x,v) and v = Fz = FT{x) (this can be interpreted as adding a feedback 
loop around the dashed box shown in Figure 5.4). We claim that system (5.14) 
has a; as an asymptotically stable equilibrium, with a potentially large domain 
of attraction: every solution £ with the property that £(£) G O for all t must 
converge to x . Indeed, suppose that £ is any solution of (5.14) which stays in 
O. Let C{t) ■= T(£(i)). Then ((t) = (A + bF)((t), so ((t) -> as t -> oo. Since 
T has a continuous inverse, this means that £(t) — > x° as t — > oo. (In addition, 
if £(0) is near x° then £(0) is near zero, so £(£) remains close to zero and thus 
£(£) remains close to a; for all t.) 

Lemma/Exercise 5.3.3 Prove that we have an equivalence relation, in the 
following sense: 

(a) For each E and open OCX, (E, O) is feedback equivalent to (E, O). 

(b) (E,0) feedback equivalent to (E,C) implies (E,C) feedback equivalent to 

(c) If (E,C) is feedback equivalent to (E,0) and (E,C) is feedback equivalent 
to (E,C), then (E,C) is feedback equivalent to (E,C). □ 

Lemma/Exercise 5.3.4 Suppose that the pairs (A, b) and (A, b) G <S„ ; i are 
feedback equivalent in the sense of Definition 5.2.1. That is, there exist T G 
GL{n), F G M lx ", and a nonzero scalar V such that T~ 1 (A + bF)T = A and 
T~ 1 bV = b. Consider the systems E and E given respectively by x = Ax + ub 
and x = Ax + ub. Show that, for each open subset O C R™, (E, O) is feedback 
equivalent to(E,T" 1 e'). □ 

The study of feedback linearization is simplified considerably by the following 
observation. For each positive integer n, we let E„ be the continuous-time linear 
system x = A n x + ub n , where A n and b n are in the controller form given by 
Equation 5.3: 



/0 1 . 

1. 


. 







0. 
\0 . 


. 1 

. oy 


b n := 






By Theorem 14 (p. 191), any single input n-dimensional controllable pair is feed- 
back equivalent, in the sense of Definition 5.2.1, to this special pair {A n ,b n ). 
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Therefore, by Lemma 5.3.4, for any n-dimensional controllable single-input lin- 
ear system £ and any open subset O C R n , (S,C) is feedback equivalent to 
(£ n ,W), for some open subset W C K n . By transitivity of equivalence, cf. 
Lemma 5.3.3, we conclude: 

Lemma 5.3.5 The system £ is feedback linearizable about the state a; if and 
only if there exist an open OCX containing x° and an open W C K n such that 
(Ti,0) is feedback equivalent to (E„,W). □ 

Observe that Equations (5.1 1)-(5.12) when specialized to £„ can be written 
equivalently as follows: 

T.(x)f(x) = A n T{x)-°^\b n (5.15) 

pyx) 

and 

T*(x)g(x) = -^-b n . (5.16) 

P(x) 

The main result to be proved in this section is as follows. We employ concepts 
from the nonlinear controllability Chapter 4. 

Theorem 15 The system (5.10) is feedback linearizable about x° if and only if 
the distribution 

An-i := A fliad/fli ... iad? - afl (5.17) 

is involutive when restricted to some neighborhood of x , and the set of vectors 
< g(x°), ad/<?(x°), . . . , ad^ _ g(x°) > is linearly independent. 

(When n = 1, the condition is simply g(x°) ^ 0.) Observe that, if the n 
vectors shown are linearly independent, then dim A„_i(:r ) = n — 1. Thus, 
dimA n _i = n — 1 on some neighborhood O of x . On any such set O, the 
involutivity condition amounts to the requirement that 

[a,dyg,a,dig)(x) e A„_i(x) for all x & O and slli,j & {0, . . . ,n - 2} (5.18) 

(cf. Lemma 4.4.93b). Equivalently, by Frobenius' Theorem 10 (p. 175), the 
condition is that A n _! must be integrable (in some neighborhood of x°). 

We need to establish a preliminary general result concerning smooth func- 
tions and vector fields. 

Lemma 5.3.6 Let O be an open subset of E", ip G F(O), f,g G V(O). The 
following properties are equivalent, for any given nonnegative integer £: 

1. For each < i < £, L g L l f (p = 0. 

2. For each < j < (., L ad i u> = 0. 



3. For each i, j > with z + j < £, L ad j L\ip = 0. 
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Furthermore, if these properties hold, then: 

For each i,j > with i + j = t + 1 , L ad e+i ip = {-l) l L adJ Liip. (5.19) 

Proof. Note that statements 1 and 2 are particular cases of 3, corresponding to 
the choices j = and i = respectively. We will prove, separately, that each of 
them implies 3. The proof will be based upon a useful identity which we derive 
first. 

Equation (4.3) says that L a dfh = LfLh — L^Lf for any two vector fields 
/,/i£ V(O). We may apply this formula, in particular, when h has the form 
ad^<7, and thus derive 

L 3.d j f +1 g = L f L a.d j f g ~ L a.djg L f ' 

This is an equality among operators on smooth functions, so in particular we 
may evaluate at any function which is a directional derivative of tp, and conclude: 

L ^ 9 L > = L f L ^ f9 L W ~ ^ad^V Vm > 0. (5.20) 

Suppose now that 1 holds. We prove 3, by induction on j. The case j = is 
just 1 (since a.d tg = g, by definition). Suppose that 3 has been shown to hold, 
for a given j (and for all i). We need to prove that, for any i, L ad j+i L\ip = if 

i + (j + 1) < £. This is a consequence of identity (5.20). Indeed, if i+ (j + 1) < £ 
then also i + j < £, so, by inductive hypothesis, L ,, L\ip = 0, and so 

f9 J 

K d ^ g L l fV = -L^Lf 1 ?. (5.21) 

The right-hand side vanishes, again by inductive hypothesis since (i+ 1) +j < £, 
so L aA i+i L\(p = 0, as claimed. 

Next, supposing that 2 holds, we prove 3, this time by induction on i. The 
case i = is just 2. Suppose 3 has been shown to hold, for a given i (and for 
all j). We must show, for any j, that L ad j L % f tp = if (i + 1) + j < I. As 

* + i 5: ^ by inductive hypothesis L ad j L^tp = 0, so Equation (5.21) again holds. 
By induction, the left-hand side vanishes, so the right-hand term does too. 
Finally, we assume that 3 is true and show 



J ad 



i+lg if = (-iyL adi+1 -, g L)cp , for i = 0, . . . ,1 + 1 (5.22) 



by induction on i. When i = 0, this is clear. Assume now that (5.22) has been 
proved for some i < £+1. We consider Equation (5.20), applied with j = £ — i. 
Since the first term in the right vanishes, because i + j = £ and 3 is true, we 
have that 



>P = - L ad5-* fl L / + V = -i Bd (/+i)-(<+i) fl i/ + V- 



Substituting into (5.22) completes the induction. 

From this Lemma, the following consequence is clear: 
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Corollary 5.3.7 Let O be an open subset of R n , ip G F(O), f,g G V(O). The 
following properties are equivalent, for any given nonnegative integer £: 

1. For all x eO, L g L)p{x) = if < i < £, and L g L* f +1 cp(x) / 0. 

2. For all x G O, L ad ; tp(x) = if < j < £, and L ad i+i ip(x) ^ 0. 

3. For all x 6 C, and for i 7 j > 0, -^ ad j L l r(f(x) = if 2 + j < f, and 

L ad> L^(a;)^0ifi + i = £+l. D 

Corollary 5.3.8 Let O be an open subset of R™, <p G ¥{0), f,g G V(O), and 
suppose that the equivalent properties in Corollary 5.3.7. Then, for each x G O 
the £ + 2 vectors 

g(x),a,d f g(x) ..., a,d e f +1 g(x) 

are linearly independent, and the £ + 2 row vectors 

V<p(x),VL t <p(x),...,VL t + 1 <p(x) 
are linearly independent. 
Proof. Consider the following matrix 

M(x) = (Vif(x),VLMx),...,VL i + 1 l p(x)y ■ (g{x)Ms9{x),...,B,A t f +1 g{x) 
of size {£ + 2) x (£+ 2), for each x G O. Its (i, j)th entry is: 
M(x) itJ = V (ijrV) (*) • adf ^(ar) . 



By (3), M(x) has the triangular structure \Z*\ for every x, where the skew- 



diagonal elements are nonzero. Thus M(x) is invertible, which proves the desired 
conclusions. I 

The main step in proving Theorem 15 is given by the following technical 
result. 

Proposition 5.3.9 Let E be a system x = f(x) + ug(x), and let O be an open 
subset of the state space X C W 1 . Assume that T = (Ti, . . . , T n ) : O — » 1" and 
a, /3 : — > R are smooth mappings, with /?(cc) 7^ for all x G 0, and are such 
that Equations (5.15) and (5.16) hold for all x G O. Then, writing tp := Ti, the 
following properties hold for each a; G 0: 

L g L)y{x) = 0, i = 0,...,n-2, L g L n f x <p{x) ± (5.23) 

—L n (p(x) 1 

T i+ i = LW i = 1, • • • , n - 1, a(x) = J , /3(x) = = (5.24) 

+ S ^ { ' L g L n f l p{x) ' { ' L g L n f\{ X y ' 

\7<p(x) . &d j f g(x) = , j = 0, . . . , n - 2 , Vcp(x) . &d n f x g{x) =£ (5.25) 

g(x), adfg(x) . . . , ad^ _ g(x) are linearly independent (5.26) 

T*(x) is nonsingular. (5-27) 
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Conversely, suppose that (5.26) is true for all x G O, and assume that p> G F(0) 
is so that (5.25) holds for all x G O. Then L g L n f ~~ 1 ip{x) ^ for all x, so 
we may define T\ := tp and T 2 , . . . ,T n ,a,/3 via (5.24); with these definitions, 
Equations (5.15) and (5.16) hold for all x G O. 

Proof. Suppose that T, a, j3 are so that Equations (5.15) and (5.16) hold for all 
x G O, and let ip := T\. Observe that T*(x)f(x) = (LfTi(x), . . . , LfT n (x))' and 
T*:(x)g(x) = (LgTi(x), . . . , L g T n (x)Y , and because of the form of the matrix 
A n , A n T(x) = (T2(x), . . . ,T n (x), 0)'. So Equation (5.15) says 

(L f T 1 (x),...,L f T n (x))' = (r 2 (x),...,T n (x),-^j . 

Comparing the first n — 1 entries gives T 2 = Lf(p, T3 = -L/T2 = L"lip, and, 
in general T i+1 = L^y>, j = 1, . . . ,n — 1, as required for (5.24), while the last 
coordinate gives Ulp = —a//3. On the other hand, Equation (5.16) gives 

{L g T 1 {x),...,L g T n {x))' = (V..,0,^y) . 

From the last coordinate, L g U l r ip = 1/(3, which together with L^ip = —a/j3 
gives the missing part of (5.24), This also says that L g Ul~ p is everywhere 
nonzero, so the last part of (5.23) holds. On the other hand, the first n — 1 
coordinates provide L g Ti = L g L l r p = for i = 1, ... ,n — 1, so all of (5.23) 
has been proved as well. 

We now show that (5.23) implies all three of (5.25), (5.26), and (5.27). The 
first of these is simply the equivalence of 1 and 2 in Corollary 5.3.7, when 
£ is chosen as n — 2, and the last two follow from Corollary 5.3.8, because 
T=(p,L f p,...,L]- 1 pY. 

We conclude by establishing the last (converse) statement. Let p G F(O) be 
given, and assume that (5.25) and (5.26) hold for all x. By the equivalence of 1 
and 2 in Corollary 5.3.7, also (5.23) holds, and in particular L g Ul~ p(x) ^ for 
all x, so we may indeed define a and (3 as in (5.24). We also let Xi := L % 7 </?,for 
i = 0, . . . ,n — 1. Comparing coordinates, both (5.15) and (5.16) are verified. I 



Proof of Theorem 15 

Suppose that £ is feedback linearizable around a; . Then, there exists some 
neighborhood O of x° and T, a, and (3 (nowhere vanishing) so that both (5.15) 
and (5.16) hold on O. Then, by Proposition 5.3.9, there exists a <p G V(O) so 
that (5.25) holds. This implies, in particular, that V '(p(x) 7^ for all x, and 
\7p(x).ad' l fg(x) = for each j = 0, . . . ,n — 2, so the distribution A„_i (which 
has constant rank n— 1, by (5.26)) is integrable, by the characterizations (4.37)- 
(4.38). So A„_i, seen as a distribution on O, is involutive. 
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Conversely, assume that A n _i is involutive, on some open set V which con- 
tains x°, and g{x°), ad/g(a; ), . . . , ad^~ g{x°) are linearly linearly independent. 
Using if necessary a smaller V, we may assume that g(x), adfg(x), . . . , adf~ g(x) 
(and, in particular, the first n — 1 vectors in this list) are linearly independent 
for each x G V. So, on V, A„_! is constant rank and involutive. By Frobe- 
nius' Theorem 10 (p. 175), A n _i must be integrable, so there exists a neigh- 
borhood O C V of x° and a tp G V(O) so that, for all x G O, Vtp{x) ^ 
and, for each j = 0, ... ,n — 2, S7(p(x).8A J fg(x) = 0. If it were the case that 
\7ip(x).a.d r l" g(x) = for some x G O, then ad^ _ g(x) would be in the span 
of g(x), a.dfg(x), . . . ,ad^~ g(x) (because these are n — 1 linearly independent 
vectors orthogonal to the nonzero vector \7ip(x)), and this contradicts the fact 
that g(x),adfg(x), . . . ,acC~ g(x) are linearly independent. Therefore (5.25) 
and (5.26) hold for all x G O, and the second part of Proposition 5.3.9 provides 
a T, as well as an a and a (nowhere vanishing) f3 so that both (5.15) and (5.16) 
hold. This provides a feedback linearization, except for the fact that T is not 
necessarily a diffeomorphism. However, the first part of Proposition 5.3.9, ap- 
plied to this T, shows that T*(x) is nonsingular. Thus, by the Implicit Function 
Theorem, we may restrict O to a smaller neighborhood of a; in such a manner 
that, in this restricted set, T indeed admits a smooth inverse. I 

Example 5.3.10 We consider a model of an arm driven by a motor through a 
torsional spring, see Figure 5.5; this is used as a simplified model of a single-link 




Figure 5.5: Flexible link. 

flexible-joint robot. The equations for the angular positions 9\ and 62 of the 
arm and shaft can be shown to be (see [378]): 

101 + MgL sin 6>i + k{6 1 - 6 2 ) = 

J9 2 - k{e 1 - e 2 ) = u, 

where / and J are moments of inertia, k is a spring constant (notice that the 
term k(8i — 9 2 ) stands for a restoring force proportional to the difference between 
the angles, that is, the torsion of the spring), M is the mass, L is the distance 
from the joint to the center of mass of the arm, and u, the control, is the torque 
being applied by a motor. We let x\ := 6\, x 2 := 9\, x% := 9 2 , and X4 := 9 2 
and view the system as a system with state space X = R 4 . (More precisely, one 
could represent this as a system whose state space is a differentiable manifold, 
accounting for the fact that 9\ — 9 2 only matters modulo 2ir.) Letting a := 
MgL/ 'I, b := k/I, c := k/J, and d := 1/ J, this is a system x = f(x) + ug(x) 
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with 

(XT. 
-asm Xl -b( Xl -Xs) | /;[r) 
c(xi - x 3 ) 
We compute 




(g{x),a,d f g{x),a,d 2 f g{x),a,d 3 f g{x)) 



(constant matrix). This shows that g(x°), a.djg(x°), a.dfg(x°), a.dfg(x°) are lin- 
early independent for each x° and, since [ad\-g, ad"Lg] = G p A3 for all i,j € 
{0, 1, 2}, that A3 is involutive. Thus, the system is feedback linearizable around 
every x . However, much more is true in this example. Let us compute p as 
in Proposition 5.3.9. Property (5.25), and the explicit form found above for the 
adWs, says that we must satisfy, for each x near any given x Q , 

dtp dtp dip dp 

dx- 4 = dx- 3 = dx- 2 =0 ^dx-^ - 

This set of partial differential equations have an obvious solution, which happens 
to be globally defined on R 4 , namely, p(x) := X\. This means that Proposi- 
tion 5.3.9 can be satisfied globally. With this choice of tp, we compute 

Lftp(x) = xi 

Lcip(x) = —as'mxi — b(xi — X3) 

Lcip(x) = —ax2Cosxi — b(x2 — X4) 

Lfip(x) = ax^sinxi + (acosxi + b)(asinxi + b(xi — X3)) + bc(x\ — X3) 

LgLrip(x) = bd . 

The obtained mapping T(x) = ((p(x),Lfip(x),Lfip(x),LfCp(x))' happens to de- 
fine a global diffeomorphism from R to K : its inverse is given by 

T~ 1 (z) = (zi,Z2,z\ + (1/6) (23 + asinzi),Z2 + (1/6) (24 + a,Z2COSZi))' . 

Thus, with j3{x) = 1/bd and a(x) = —(l/bd)L'iip(x), one obtains a global 
feedback linearization. □ 

Exercise 5.3.11 Property 5.23 in Proposition 5.3.9 says that E must have 
relative degree n with respect to tp on the open set O, and in particular at the 
point x°. In general, for any system (E) x = f{x) + ug(x), together with a 
smooth tp : X — > R (which we think of as an "output map" for E), and a state 
x , one defines the relative degree of (E, tp) at x° as the smallest positive integer 
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r, if one exists, with the property that L g L\tp{x) = for each < i < r — 2 and 
each x in some neighborhood O of x° (this condition is vacuous if r = 1), and 
LgVr tp(x°) y£ 0. If for every i > it is the case that L g L\tp[x) = on some 
neighborhood of a; , we will say that the relative degree exists but is infinite. 

(a) Show that, if (£,</?) has a finite relative degree r around x°, then there 
are real numbers ^o, • • • , Mr, an d v ^ 0, with the following property: For each 
continuous input u defined on some interval [0, t], r > 0, let x(t) = <f>(t, 0, x° , u) 
be the solution of x(t) = f(x(t)) + u(t)g(x(t)), which is defined on some maximal 
interval [0,e), e > 0, and let y(t) = tp{x{t)). Then y is class C r , and its 
derivatives at i = satisfy y^(0) = ^, for j = 0,...,r — 1 and 2/ r '(0) = 
fi r + i>u(0). Note that this says that the first r — 1 derivatives of the "output" 
y are independent of the input, but the rth derivative does depend on u. 

(b) Show that for a controllable linear system x = Ax + ub with (p(x) = ex 
linear and c/ 0, a finite relative degree is always defined, it is < n, and it is 
the smallest r so that cA r ~ 1 b ^ 0. 

(c) Give an example of E, tp, x° for which the relative degree does not exist. To 
make the example interesting, you are not allowed to pick one where the relative 
degree exists but is infinite. □ 

Exercise 5.3.12 Suppose that a; is an equilibrium state: f(x) = 0. Show 
that, if the system is feedback linearizable about x°, then it is possible to choose 
T{x") = and a{x a ) = 0. □ 

Exercise 5.3.13 Suppose that x° is an equilibrium state: f(x°) = 0. Prove 
that the linear independence condition in the statement of Theorem 15 is equiva- 
lent to asking that the pair (A, b) be controllable, where (A, b) is the linearization 
of E about x = x° and u = 0. (That is, A = f*(x°) and b = g(x ).) □ 

Exercise 5.3.14 Suppose that x° is an equilibrium state: f(x°) = and the 
system E has dimension 2. Show that E is feedback linearizable about x° if and 
only if the linearization of E about x = x° and u = is controllable. □ 

Exercise 5.3.15 Give an example of a E of dimension 3, and an equilibrium 
state x°, such that the linearization of E about x = x° and u = is controllable 
but E is not feedback linearizable about a; . □ 

Exercise 5.3.16 Check that the system x\ = sina;2, £2 = x\ + u is feedback 
linearizable about x° = (0,0)'. Next, find explicit expressions for T, a, and f3, 
so that both (5.15) and (5.16) are verified for all x G O := R x (-tt/2, tt/2). □ 

Exercise 5.3.17 Show that if the conditions of Theorem 15 hold, then the 
distribution A^ := A ad a d fc_1 ^ s integrable, on some neighborhood of x a , 

for each k = l,...,n — 2. (Hint: For each Afe, show that tp = L J f ip, j = 
0, . . . ,n — k — 1 provide independent integrals, if tp is as in the proofs.) □ 
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Exercise 5.3.18 O Show that £ is feedback linearizable in the restricted sense 
that one may pick (3{x) = 1 if and only if, in addition to the conditions of 
Theorem 15 (p. 200), one has that 

I &d)g, dA 3 f g\ g p A k , k = 0, . . . , n - 1 , j = 0, . . . , k - 1 

(on some neighborhood of x ). □ 

Exercise 5.3.19 O A far more restrictive problem is that of asking that E 
be linearizable by means of coordinate changes alone, i.e., that there be some 
diffeomorphism defined in a neighborhood of x , and a controllable pair (A, b), 
so that T*(x)f(x) = AT(x) and T*(x)g(x) = b. This can be seen as feedback 
linearization with a = and (3 = 1. Show that such a linearization is possi- 
ble if and only if g(x ), a.dfg{x°), . . . , ad^" g(x ) are linearly independent and 
[adjg, ad^g] = for all i,j > 0. □ 

5.4 Disturbance Rejection and Invariance* 

A large amount of the linear systems literature has been devoted to the study 
of synthesis problems in which a feedback law u = Fx is sought to achieve 
objectives other than — or more often, in addition to — stabilization or pole 
assignment. 

This section describes one of these objectives, the "disturbance rejection" 
(sometimes also called "disturbance decoupling") problem. The method of so- 
lution illustrates the application of what is sometimes called the (linear-) "geo- 
metric approach" to linear control. This approach is based on the use of certain 
spaces that are invariant under the A matrix, modulo the action of controls. Il- 
lustrating the use of these spaces is our main purpose here; the material will 
not be used in later sections. 

Still K denotes an arbitrary field, and unless otherwise stated, (A, B) is an 
arbitrary pair, A g K™ x ", B g K™ xm . For the case of no controls, B = 0, the 
following reduces to the classical notion of invariant subspace. We let B = col B, 
i.e. im B when the latter is thought of as a map 

B : U = K m -» X = K n . 

Definition 5.4.1 A subspace V of X = K n is ^4-invariant mod B if 



AVCV + B 



One says equivalently that V is (A, B) -invariant. □ 

Since the concept does not depend on B itself but only on its image B, 
the alternative terminologies A-invariant mod B and (A, £?)-invariant are also 
used. 



* This section can be skipped with no loss of continuity. 
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Exercise 5.4.2 If V is ^-invariant mod B and if B C V then V must also 
contain the controllable subspace TZ(A, B). □ 

By definition, V is (A, _B)-invariant if and only if for each x g V there exist 
a z g V and a u g IX such that 

Aa; = 2 + -Bit . 

For a discrete-time system with matrices (A, B), this implies that for each state 
a; in V there is another state z in V such that i^z (use — u as a control), and 
hence, by induction, the state may be kept in V for arbitrary long times. An 
analogous statement holds for continuous-time systems; the following observa- 
tion is useful in establishing it: 

Lemma 5.4.3 The subspace V of X is A-invariant mod B if and only if there 
exists a matrix F g K mx ™ such that V is (A + Si 7 ')-invariant. 

Proof. Sufficiency is trivial. To prove necessity, let {v±, . . . ,v r } be a basis of 
the ^-invariant mod B subspace V. First choose for each i some m, g 11 such 
that 

Avi + But g V . 

Now define on V 

Fvi := u t 

and let F be arbitrary on a complement of V. I 

The matrix F is interpreted as a feedback transformation u = Fx. 

Exercise 5.4.4 Let S be a linear time-invariant continuous-time system (A,B). 
Prove that the subspace V is ^-invariant mod B iff for each x g V and each 
r > there is some u> g Xv ^' such that 

£(£) g V for all ig [0,r] 

if £ is the resulting path ip(x, lo). □ 

The above concept of relative invariance arose originally when treating the 
following problem. For concreteness we deal with continuous-time systems, but 
the discrete-time case is entirely analogous. 

Consider linear time-invariant continuous-time systems with outputs, over 
K, with m + I inputs partitioned as (u, v): 

x = Ax + Bu+Ev y = Cx, (5.28) 

where v is to be interpreted as a "disturbance." This disturbance cannot be 
measured directly by the controller, and the objective is to design a feedback 
law 

u = Fx + v! 
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such that v has no effect whatsoever on the output, no matter what v! or the 
initial condition x° of (5.28) are. 
For instance, in the system 



Xl 


= 


x 2 + u 


%2 


= 


V 


y 


= 


Xl 



the disturbance v is not decoupled from the output — in particular, for x° = 
and u = one has y = v. But using the feedback transformation 

u = —X2 + v! 

there results a system for which 

y (t) = x\ + / u'{s)ds 
Jo 

is independent of v. 

In general terms, then, we wish to have 

y (t) = Ce tA x Q + C f e {t -^ A {Bu'{T) + Ev{t)) dr 



independent of v, for some matrix of the form 

A = A + BF . 
This is equivalent to the requirement that 

C f e^- T ^ A Ev{T) dr = 
Jo 

for all (measurable and essentially bounded) v : [0,T] — > R , all T > 0, and 
all t £ [0, T]. Considering this expression as the i/o behavior of the system 
(A,E,C) with zero initial state, we see from Lemma 2.7.13 that the problem 
becomes: 

Find a matrix F so that C{A + BF) l E = for all i > 0. (DRP) 

In principle, the problem of deciding whether there is such an F (and finding 
one in that case) is highly nonlinear in terms of the data A, B,C,E; however, 
the following Lemma transforms it into a geometric question. Let £ := imE. 

Lemma 5.4.5 The DRP problem is solvable if and only if there exists a sub- 
space V C R" so that 

£ C V C ker C 

and 

V is A— invariant mod B . 
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Proof. Assume that there is such a subspace V. By Lemma 5.4.3 there exists 
some F so that V is A + £>_F-invariant. Thus, for all i > 0, 

(A + BFy£ C (A + BFyV C V C kerC, 

and therefore, C(A + BF) l E = 0, as wanted. Conversely, given such an F, it is 
only necessary to define 

V := K{A + BF, E) , 

which has the desired properties. I 

Deciding whether there is such a subspace turns out to be easy, once another 
concept is introduced. For any given (A, B,C), we consider the set of possible 
(A, _B)-invariant subspaces V which are included in ker C. There always is at 
least one such subspace, since V = is (A, £>)-invariant. We order these sub- 
spaces by inclusion. If Vi and V2 are two maximal spaces with these properties, 
then their sum is again an (A, i?)-invariant subspace included in kerC, so, by 
maximality, we have that 

Vi = Vi + v 2 = v 2 . 

In other words, there is a unique such maximal (A, i?)-invariant subspace in- 
cluded in ker C. This is denoted 

V*{A,B,C). 

Furthermore, any V which is (A, _B)-invariant and satisfies V C kerC must be a 
subspace of V*(A, B, C), by the same argument. So if any such space contains 
£, V* does also. Thus, we have proved the following result. 

Theorem 16 The DRP problem is solvable iff£C V*{A,B,C). □ 

The space V* can be calculated recursively, as shown by the next problem. 
On a computer, it is necessary then only to solve a sequence of linear equations 
in order to find this space, and a further Gaussian elimination can be used to 
check the property that £ C V*(A, B, C). 

Exercise 5.4.6 Consider the following algorithm: 

V° := kerC, 

and, for i = 1 , . . . , n, 

V 1 := kerCpl^OS + F*- 1 ). 

Prove that V n = V*(A,B,C). □ 

Remark 5.4.7 For motivation regarding the DRP problem, the reader might 
want to review the material on PID control in Chapter 1. The objective there 
was somewhat weaker than here, however, in that it was only required that 
the effect of disturbances be asymptotically damped, as opposed to completely 
canceled. Such a more realistic objective can also be studied with geometric 
techniques. We picked cancellation merely as the simplest illustration of these 
tools. □ 
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5.5 Stability and Other Asymptotic Notions 

In studying stability, we restrict attention to time-invariant systems only. Time- 
varying system stability is an important area, and many results that are rela- 
tively easy to establish in the time-invariant case become much more difficult 
and interesting in the general time-varying case. However, the main system 
theoretic ideas are already illustrated by the case we treat here, while many 
technical complications can be avoided. The objective here merely is to provide 
a basic introduction to these ideas in the context of control problems. 

From now on in this chapter, and unless otherwise stated, system means 
time-invariant topological system in the sense of Definition 3.7.1, that is, we 
ask that trajectories depend continuously on the initial state. This provides the 
right generality for establishing many of the basic facts. Readers interested only 
in continuous-time systems may simply substitute "continuous-time system" for 
"topological system," but this does not simplify the proofs in any substantial 
manner. When dealing with stability notions for continuous-time systems and 
for linear systems, unless otherwise stated we will assume we are working over 
K = R. 

The next definition is an infinite-time version of that in Definition 3.7.5. 

Definition 5.5.1 Let y,z € X, and assume that V is a subset ofX containing 
both y and z. Then, z can be asymptotically controlled to y without 

leaving V if there exists some control v € W ' 00 ' admissible for z so that: 



For the path (" := tp(z, v), 



lim CO) = y 

t— »oo 



• ((t) e V for allt e T+. 

When V = X, one just says that z can be asymptotically controlled to y. □ 

Definition 5.5.2 Let E be a topological system and x° an equilibrium state. 
Then E is: 

• Locally asymptotically controllable to a; if for each neighborhood V 
of x° there is some neighborhood W of x° such that each x € W can be 
asymptotically controlled to x° without leaving V . 

• Globally asymptotically controllable to x° if it is locally asymptoti- 
cally controllable and also every a; € X can be asymptotically controlled to 
x°. 

For systems with no control, the more standard terminology is to say that the 
system E is (locally or globally) asymptotically stable with respect to x° , or 
that x° is an asymptotically stable state for the system. □ 
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For linear systems, and unless otherwise stated, we always take x° = when 
considering asymptotic controllability (or stability). 

We use also the term asy controllable instead of asymptotically controllable. 
The interpretation of E being asymptotically controllable is that it should be 
possible to drive states close to the desired x° (or arbitrary states, in the global 
case) into x Q without large excursions. 

Asycontrollability is a necessary condition for feedback stabilizability, in the 
following sense: 

Lemma/Exercise 5.5.3 Consider the continuous-time system E: 

x = f(x,u) 
of class C 1 and let x° G X. Assume that there exists some function 

of class C 1 so that x° is a local (respectively, global) asymptotically stable state 
for the "closed-loop system" 

x= f(x,k(xj) . 

Then £ is locally (respectively, globally) asymptotically controllable to a; . The 
analogous result holds for discrete-time systems of class C°. □ 

Remark 5.5.4 One could consider also a weaker notion, dropping the large 
excursion part and requiring only that states close to x° be asymptotically 
controllable to x°. For systems with no control, the standard terminology is then 
that a; is an attractor. For linear systems attractivity results in an equivalent 
definition (see below). □ 

Proposition 5.5.5 Consider the differential equation 

x = Ax 

where A £ W nxn , thought of as a continuous-time linear system E with no 
controls. Then, the following statements are equivalent: 

• E is locally asymptotically stable. 

• E is globally asymptotically stable. 

• A is Hurwitz. 

Analogous statements hold for the discrete-time system x + = Ax and the dis- 
crete-time Hurwitz (convergent) property. 
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Proof. First we observe that the local notion implies the global one in the 
linear case. Indeed, if there exists some 5 > so that each solution with initial 
condition \\z\\ < 5 converges to the origin, then for any initial condition y ^ 0, 
letting 

_ 5 

it follows that e z — > 0, and therefore also that 

o 

Since all solutions of x = Ax converge to zero if and only if A is Hurwitz, 
by Proposition C.5.1 in Appendix C.5 on stability of linear equations, it is only 
necessary to prove that, if A is Hurwitz, then for each e > there is a 5 > 
such that ||£(£)|| < e for all t > whenever ||£(0)|| < 6. Consider the function 

a (t) := \\e tA \\ ,t> 0. 

Since A is Hurwitz, a(t) — > as t — > oo, so by continuity of a, it must be 
bounded, a(t) < K for all t. Since 

u(t)\\<^t)\\m\\<KU(o)\\, 

we may pick 6 := e/K . The discrete-time case is entirely analogous. I 

Proposition 5.5.6 Let E = (A, B) be a continuous-time (respectively, dis- 
crete-time) time-invariant linear system. The following statements are then 
equivalent: 

1. E is globally asymptotically controllable (to 0). 

2. E is locally asymptotically controllable (to 0). 

3. Each root of \ h as negative real part (respectively, has magnitude less 
than 1). 

4. There is some F € R mx ™ such that A + BF is Hurwitz (respectively, 
discrete-time Hurwitz). 

Proof. We only prove the continuous-time case, the discrete-time case being 
analogous. As in the proof of the previous lemma, the local and global properties 
(1) and (2) are equivalent, since controlling z asymptotically to zero using the 
control cu is equivalent to controlling 

2IMI 
6 

using the scaled control 

2 ||2/|| 



-u) . 
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Assume now that (1) holds. Pick any 

z 2 6 R n - r 

where the integer r is as in the Kalman controllability decomposition, Lemma 
3.3.3, and consider 

x:=t(°' 

where T is the similarity used in that Lemma. By assumption, there is some 
control u! on [0,oo) such that £(£) — > 0, where £ := ip(x,u>). It follows that also 
£(£) := T" 1 ^) converges to zero. But £(t) has the block form 

with 

6(*) = 4$C2(*) , c 2 (o) = ^ 2 . 

Since Z2 was arbitrary, this means that A3 in Lemma 3.3.3 must be Hurwitz, so 
(3) is proved. 

That (3) implies (4) is a consequence of Theorem 13 (Pole-Shifting), since 
the polynomial \ in the statement of that Theorem can be made Hurwitz by 
an appropriate choice of the feedback F. 

Finally, (4) implies (2) because of Lemma 5.5.3; explicitly, by Proposition 
5.5.5, x = (A + BF)x is globally asymptotically stable. Given any x € X, 
consider the solution £ of this closed-loop system with £(0) = x. The control 

w(t) := F£(t) 
is then as desired for global asymptotic controllability. I 

Thus, in the linear case we may check that £ is asymptotically controllable 
in two steps: First obtain the decomposition in Lemma 3.3.3, and then check 
that A 3 is Hurwitz. Since global and local notions coincide, we talk in the linear 
case just of asymptotically controllable or asymptotically stable systems. 

The equivalence of (4) with £ being global asymptotically controllable is a 
characteristic of linear systems. Because of this equivalence, one often refers to 
asymptotically controllable linear systems as stabilizable systems. For nonlinear 
systems, being asymptotically controllable is not in general equivalent to the 
existence of a stabilizing feedback law u = k(x), at least if k is required to be 
smooth or even continuous; one exception is the local result presented later, 
which applies if the linearization at the origin is itself already asymptotically 
controllable. See Section 5.9 below for further discussion of this point. 

There is an analogue of the Hautus controllability condition, Lemma 3.3.7, 
that applies to asycontrollability: 
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Exercise 5.5.7 Prove that the continuous-time (respectively, discrete-time) 
linear time-invariant system £ = (A,B) is asymptotically controllable if and 
only if 

rank [XI - A, B] = n 

for all A € C with nonnegative real part (respectively with magnitude greater 
than or equal to 1). □ 

Exercise 5.5.8 State and prove results analogous to Proposition 3.3.13 for the 
distance to the sets of non-asymptotically controllable discrete-time or contin- 
uous-time time-invariant linear systems. □ 

Exercise 5.5.9 O Let <S^'^ H , respectively <S°'^, denote the set of all control- 
lable pairs (A, B) 6 iSJ m for which A is discrete-time Hurwitz (convergent), 
respectively Hurwitz, and consider the map 

: (A, B) -» ((A - I)(A + I)' 1 , (A + I^B) . 

Prove that this map induces a bijection between <S^'„ H and <S£'^. Hint: Use 
the Hautus criterion to check controllability of f3(A, B). □ 

5.6 Unstable and Stable Modes* 

The conditions for a linear time-invariant system to be asymptotically control- 
lable can be restated in various manners. In this Section we prove a result that 
can be read as saying that a system is asymptotically controllable if and only if 
its unstable modes are controllable. For simplicity, we let K = R, though many 
of the results could be stated in far more generality. Accordingly, "polynomial" 
will mean polynomial with real coefficients. 

Assume that x = x( z ) ls a polynomial that factors as 

X — X g X b 

in such a way that \ an d X h (read "the good part" and "the bad part" of \) 
have no common complex roots. By the Euclidean algorithm, we know then 
that there exist two polynomials p and q such that 

px g + qx b = 1 , 

and therefore, by specialization, any square matrix C satisfies 

I = p(C) Xg (C) + q(C) Xb (C) , (5.29) 

where / is the identity matrix of the same size as C. Further, the product 
decomposition of \ gives that x(C) = \ (C)x h (C) = X h (C)x (C)i so the impli- 
cation 

X (C) = =* x b (C)x g (C) = (5.30) 

holds. 



* This section can be skipped with no loss of continuity. 
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Lemma 5.6.1 For any decomposition into two polynomials x h an d X as above, 
and any matrix C for which x(C) = Oj 

imx g (C) = kerx h {C). 
Proof. If x h {C)x = 0, then it follows from (5.29) that 

x = p{C)x g {C)x = Xg {C)x , 
where x := p(C)x. Conversely if x = x (C)j/, then 

X b (C)x = x b (C) Xg (C)y = 

from (5.30). I 

The Lemma applies in particular when x is the characteristic polynomial of 
C. 

We will say that a polynomial x is a Hurwitz polynomial if all its roots 
have negative real part. The characteristic polynomial of C is Hurwitz if and 
only if C is a Hurwitz matrix. Similarly one can define convergent (discrete- 
time Hurwitz) polynomials. 

Recall that the minimal polynomial mc = mc{z) of a matrix C is the 
(unique) monic polynomial of smallest possible degree that annihilates C, that 
is, 

m c (C) = 0, 

and that mc divides any polynomial which annihilates C. Since also the roots of 
the minimal polynomial are (in general with different multiplicities) the eigen- 
values of C, the following holds: 

Lemma 5.6.2 The matrix C is Hurwitz (or convergent) if and only if there is 
some Hurwitz (respectively, convergent) polynomial x so that x(C) = 0- ^ 

Given any matrix A, we now decompose its characteristic polynomial x as 
X X h by letting x collect all of the roots with negative real part. If discrete-time 
systems are of interest, one picks a decomposition into roots with magnitude less 
than one and magnitude > 1. That x (and hence also the quotient x = x/x ) 
has real coefficients follows from the fact that complex conjugate roots appear 
in pairs. 

For a linear system (A,B), we have introduced at this point two different 
factorizations of the characteristic polynomial of A: 



A c A u A g A b 



one corresponding to the Kalman controllability decomposition (and depending 
on B) and the other to stability (independent of B, but dependent on the notion 
of stability, such as discrete-time or continuous-time, being used). 
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In terms of these polynomials, the equivalence between (1) and (3) in Propo- 
sition 5.5.6, which states that the system is asymptotically controllable if and 
only if each root of x has negative real part, can be restated as: 

y divides y 

/v -u /x *g 

as a polynomial. On the other hand, the following property is easy to establish 
for the Kalman controllability decomposition: 

Lemma/Exercise 5.6.3 Let \ be any polynomial, and pick any pair {A, B). 
Let ^.3 be any matrix as in the controllability decomposition (3.6) in Chapter 
3. Then, 

im X {A)cn{A,B) «■ X {A 3 ) = 0. D 

We have the following characterization of asymptotic controllability: 

Proposition 5.6.4 Let (A, B) be a time-invariant continuous-time or discrete- 
time linear system. The following properties are equivalent: 

1. (A, B) is asymptotically controllable. 

2. im Xg (A)CK(A,B). 

3. ker Xb (A)CK(A,B). 

Proof. That the last two properties are equivalent follows from Lemma 5.6.1. 
By Lemma 5.6.3, we must prove that the system is asymptotically controllable 
if and only if 

Xg (A 3 ) = (5.31) 

for some ^3 as in that Lemma. But if (5.31) holds, then Lemma 5.6.2 implies 
that A 3 is Hurwitz (or convergent, in the discrete-time case), and so the system 
is indeed asymptotically controllable. Conversely, if the system is asymptotically 
controllable, then y_ , the characteristic polynomial of ^3, divides x , from which 

u g 

(5.31) holds because of the Cayley-Hamilton Theorem. I 

The space imx (A) = kerx,(^4), or more precisely the kernel of the cor- 
responding complex mapping, can be thought of as the set of unstable modes 
of the system, as in the next exercise. Thus, the above result can be restated 
roughly in the following way: 

A linear system is asymptotically controllable if and only if its un- 
stable modes are controllable 

which is an intuitive fact: Stable modes decay by themselves, and unstable 
modes can be controlled to zero. 



Exercise 5.6.5 Let v be an eigenvector of A. Prove that v corresponds to an 
eigenvalue with nonnegative real part if and only if it is in the kernel of X h {A) 
seen as a complex matrix. □ 



218 5. Feedback and Stabilization 

The following exercise makes use of the concept of (A, 5)-invariant subspace 
introduced in Section 5.4. It deals with the question of deciding, for the time- 
invariant continuous-time linear system 

x = Ax + Bu , y = Cx , 

whether there is any feedback law u = Fx so that 

C£(t) ^ as t -» oo 

for every solution of 

x = {A + BF)x. 

In other words, the state may not be stabilized, but the observations should be. 
When C = I, this is just the stabilization problem, and asymptotic controlla- 
bility is necessary and sufficient. More generally, the result of the exercise can 
be interpreted as saying that unstable modes must be either controllable or not 
"observable" by C . 

Exercise 5.6.6 O Show that the above problem is solvable if and only if 

kerx b (^) C K{A, B) + V* {A, B, C) . U 

5.7 Lyapunov and Control-Lyapunov Functions 

For nonlinear systems, the best one often can do in order to establish that a 
system is asymptotically stable or asymptotically controllable is to use the Lya- 
punov function method. The intuitive idea behind this method is that, if there 
exists some sort of "energy" measure of states such that this energy dimin- 
ishes along suitably chosen paths, then the system can be made to approach a 
minimal-energy configuration. 

We first illustrate the basic ideas of the Lyapunov direct method with the 
simplest (and most classical) case. Assume that X is an open subset of R™, / is 
a vector field defined on X, and V : X — > R is a differentiable function. Consider 
the new function 

LfV-.X^R, (L f V)(x) := VV(x)f(x) 

(we drop the parentheses and write simply LfV(x); also, we often insert a dot 
and write "VV(a;) • f{x) n in order to help readability). This is the directional, 
or Lie, derivative of V in the direction of the vector field / (also considered in 
Section 4.1). It is the expression that appears when taking the derivative of V 
along a path of the system with no controls x = f(x) and state space X. That 
is, if £ is any path of x = f{x), then 

^f" - L, Vm) . 
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The Lyapunov stability method is based on the following observation: If there 
is a function V such that 

LfV(x) < 

for all nonzero x, then V(£(£)) must decrease. If in addition one can guarantee 
that V{x) — > implies x — > 0, then this means that £(£) — > 0. The main point of 
this is that there is no need to compute explicitly the solutions of the differential 
equation (hence the term direct method). On the other hand, a certain amount 
of ingenuity and/or physical intuition is needed in order to come up with a 
suitable "energy" function V. 

We will prove the basic results for discrete as well as continuous-time sys- 
tems, and in fact, with the appropriate definitions, for arbitrary topological sys- 
tems. Moreover, we give a control-theoretic version of the results. To avoid con- 
fusion with the more classical term, we use the terminology "control-Lyapunov 
function" for systems with controls, and "Lyapunov function" for systems with- 
out controls. 

Definition 5.7.1 A local control-Lyapunov function for the system E 

(relative to the equilibrium state x°) is a continuous function V : X — > M. for 
which there is some neighborhood O ofx° such that the following properties hold: 

1. V is proper at x° , that is, 

{xe%\ V{x) < e) 
is a compact subset of O for each e > small enough. 

2. V is positive definite on O: 

V{x") = , and V(x) > for each x e O, x / x° . 

3. For each x ^ x° in O there is some time a G T, a > 0, and some control 
LO € li^ la '' admissible for x such that, for the path £ = ip(x,u>) correspond- 
ing to this control and this initial state, 

V(£(t)) < V(x) for all te [0,a) 

and 

V(S(a))<V(x). 

A global control-Lyapunov function for E (relative to x° ) is a continuous 
V which is (globally) proper, that is, the set 

{xeX\ V(x) < L} 

is compact for each L > 0, and such that (2) and (3) are satisfied with O = X. 

For systems without controls, we say simply (local or global) Lyapunov 

function. □ 
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Remark 5.7.2 Properness amounts to the requirement that V~ X (C) be com- 
pact whenever C C M. is compact. When X = M™, a continuous V : R" — > K>o is 
proper if and only if V~ l (C) is bounded whenever C is bounded, or equivalently, 

lim V(x) = oo . 

||a:|| — »oo 

A V which satisfies this last property, equivalent to properness in the special 
case X = K ra , is also said to be weakly coercive or radially unbounded. □ 

Local control-Lyapunov functions are often only specified in a neighborhood 
of x° . If the state space is well-behaved enough, they can be extended while 
preserving the desired properties. The next exercise illustrates this. 

Exercise 5.7.3 Let V : O — > K be a continuous and positive definite function 
defined on some neighborhood O of a; , where X = 1". Assume that V satisfies 
property (3) in Definition 5.7.1. Show that, then, there exists a local control- 
Lyapunov function W for the same system, with W = V on some neighborhood 
of x° . (Hint: For S > small enough, let W = V on the ball Bs(x°) and 
W(x) := V(S(x — x )/ \\x — x°\\) outside. Verify the definition of properness for 
e less than vai{V(x), lla; — x°|| = 5}.) □ 

The main result is as follows: 

Theorem 17 If there exists a local (respectively, global) control-Lyapunov func- 
tion V forYi, then E is locally (respectively, globally) asymptotically controllable. 

Proof. Assume that V is a local control-Lyapunov function. Choose an a® > 
so that {x | V(x) < ao} is a compact subset of O (property (1) in Definition 
5.7.1). We first remark that, if £ is a path on [0, oo) such that 

V(t;(t))->0 as i^oo, 

then, necessarily, 

£(i)^*°. 

This fact is, in turn, a consequence of the following observation: 

Claim 1: For each open neighborhood W of x° there is some (3 > such 
that the set 

{x | V{x) < 13} 

is included in W. Indeed, if this were not the case, there would exist a sequence 
{x n } of elements of X with the properties that x n <£ W for all n but 

V'(xn) — > as n — » oo . 

Without loss of generality, we assume that all x„ are in the set 

X:=>V c p|{x| V(a;) < a } 
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(we denote the complement of a set S as S c ), which is itself compact. So {x n } 
has a convergent subsequence in K, 

■Erik y *^0 5 

and by continuity of V, it follows that V(xo) = 0, so by positive definiteness 
Xq = x° . But then a; is in K, contradicting the fact that it is in W. This proves 
the claim. 

For any x G X, we say that z G X is nicely reachable from x if there exist a 
it > and a u> G W '' 7 ' such that for the trajectory £ := ip(x,u>) it holds that 

£(*) = *, 

^(<))<^(a;) for alUG (0,cr), 
and 

y(^)) < v(x) . 

Property (3) in the definition of control-Lyapunov function says that for each 
state y G 0,y ^ x° there is some state nicely reachable from y. Note also that, 
if z is nicely reachable from x, and if y is nicely reachable from z, then y is also 
nicely reachable from x. For each x G O, we let 

-B(x) := inf{V(z) | z is nicely reachable from x} . 

By the above remarks, necessarily B(x) < V{x) if x ^ x° . 
Claim 2: 

V{x) < a a => B(a;) = 0. 

Indeed, suppose that there is some x (necessarily ^ x ), with V{x) < a$ but 

B(x) = a > 0. 

Because of the choice of c<o, necessarily x G O. Let {z n } be a sequence of 
elements nicely reachable from x such that 

V(z n ) -> a 

monotonically as n — > oo. Thus, all the z n are in the compact set 

C :={z\ V(z) <V(x)}, 

and, without loss of generality, we may assume then that the sequence is con- 
vergent, 

z n — » z, V(z) = a. 

Since a ^ 0, also z ^ x°. Further, since 

a < V(x) < a , 

also z G O, so there is some y that is nicely reachable from z. We pick an e > 
such that both 

V(z) < V{x) - e and V(y) < V(z) - e (5.32) 
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and we let v € V} ' a ' be admissible for z and such that the path £ := ip(z, v) is 
so that ((a) = y and 

V(((t)) < V(z) for all te (0,cr]. 

Since V is uniformly continuous on the compact set C, there is a 5 > such 
that 

\V(a) — V(b)\ < e whenever d(a, b) < 6 and a, b e C . 

Using that £ is a topological system, we know then that there is some integer 
N such that v is admissible for zjy and such that 

doo(CAf,C) < S, 

where Cn is the path ip(zN, v). So from the choice of 5, it follows that 

\V(CN(t))-V(((t))\<e (5-33) 

for each t G [0, a\. We claim that the state y^ '■= Cn(c) is nicely reachable from 
x; this will give a contradiction, because 

\V(y N )-V(y)\<e 

together with 

V(y)<B(x)-e 

imply that V{yj^) < B(x), contradicting the definition of B(x). Concatenating 
with v a u> that (nicely) controls x to zjq, it is only necessary to prove that 

V(C, N {t)) <V{x) forallt, 

but this is a consequence of (5.33) and (5.32). The contradiction is due to the 
assumption that a = B(x) > 0. This establishes Claim 2. 

We next claim that, if V(x) < 0:0,2; / x° , then there is a sequence of states 

{x n ,n > 0} 

such that Xo = X, an increasing sequence of times t n G T + , to = 0, and controls 

such that t n — > 00 as n — > 00 and so that for each n > the following properties 
hold: 

1. lu u is admissible for x n . 

2- (f>{t n +i,t n ,x n ,u) n ) = x n +i- 

3. With £ n := V(arn,w„), ^(^(*)) < 2-"^(a;) for all t e [t n ,t„+i]. 



5.7. Lyapunov and Control-Lyapunov Functions 223 

Assume for a moment that the claim is proved. Then, we may define a control lu 
on [0, oo) as the concatenation of the w„'s. This control is admissible for x. This 
is verified by checking that each restriction to an interval [0,r) is admissible: 
Since t n —* oo, there is some n so that t„ > r; thus, the restriction to [0, r) 
is also the restriction of a finite concatenation of the u> n and hence is indeed 
admissible. Let (£, u>) be the corresponding trajectory; since £ is a concatenation 
of the £ n , it follows that 

V($(t)) ^ as t -> oo 

and therefore also that £(£) — * a; . 

To prove the claim, it will be sufficient by induction to show that for each 
x G X for which 

< V(x) < a 

there is some a > 1, and some control 10 of length a, admissible for x, such that 
the path £ := -ip(x,Lu) satisfies 

V(£(t)) < V{x) for all t 

and also 

V(i(a))<(l/2)V(x). 

By continuity of 7 at i°, there is some e > such that 

V(z) < (l/2)V(x) whenever d(z,x°) < e. 
Let u>° G 111 ' 1 ) be a constant control such that 

(j){t,0,X a ,LU ) =X° . 

Since £ is topological, there is some 5 > such that, for each y € X with 
d(y,x°) < 5, necessarily w is admissible for y and the path £ := ip(y,uP) 
satisfies d(C{t),x°) < e for all t, and hence by the choice of e so that 

V(C(tj) < (l/2)V(x) 

for all t G [0,1]. Because of Claim 1, there is some 5o > with the property 
that 

V(y)<5 =* d(y,x°)<<5. 

Now use the fact that B(x) = 0. There is, then, some r > and some control 
uj\ £ \l^°' T > such that the path £i := tp(x,u!i) satisfies that 

V(£i(t)) < V{x) for all te [0,r] 

and so that V(£,i(t)) < 6q; hence, because of the choice of 5, also 

d(^(T),X°)<S. 
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The concatenation of u)\ and (a translate of) uj° is then as desired. 

Now we prove that £ is locally asymptotically controllable. Let V be any 
neighborhood of x°. Choose some ai > such that 

{y\V(y)< ai }CV, 

and let 

a := minjao, «i} • 

We take 

W := {y | V(y) < a} 

in the definition of locally asymptotically controllable. For any y £ W,y 7^ x , 
the previous claim applies, because V(y) < ao- Thus, there is some trajectory 
(£,cj) on [0,oo) such that £(0) = y, 

t;(t) — » x as t — > 00 , 

and 

^(CW) < ^(y) < «i for all t. 

In particular, £(£) g V for all t. 

Assume finally that V is a global control-Lyapunov function, and pick any 
y £ X. Let [3 := V(y). The only fact about O and o;o used in the above proof, 
besides property (3) in the definition of control-Lyapunov function, was that 
the set 

{y I V(y) < a } 

is compact and is contained in O. So here we may take in particular O := X 
and ao := fi + 1. Pick, in the previous paragraph, V := X. Then ai := /? + 1 
can be used, and so with W as above we have that y £ W and hence that j/ is 
asymptotically controllable to i°. I 

The question of establishing converses to Theorem 17 is a classical one, not 
studied in this text; see the discussion at the end of the chapter. Property (3) 
in the definition of control-Lyapunov function (V decreases along appropriate 
trajectories) cannot be checked, for continuous-time systems, without solving 
differential equations. As discussed in the introduction to this section, Lyapunov 
analysis relies on a direct criterion for decrease, expressed infinitesimally in 
terms of directional derivatives of V along control directions, as follows. 

Lemma 5.7.4 Let E be a continuous-time system and V:l^Ka continuous 
function. Assume that O C X is an open subset for which the restriction of V 
to O is continuously differentiable and properties (1) and (2) in the definition 
of control-Lyapunov function hold. Then, a sufficient condition for V to be a 
control-Lyapunov function is that for each x £ 0,x 7^ x Q , there exist some 
u £ U such that 

VV(x).f(x,u) <0. (5.34) 

If this property holds with O = X, V is a global control-Lyapunov function. 
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Proof. We need only to establish property (3) in the definition. Pick any 
x G O, x 7^ a; , and any u G IX such that equation (5.34) holds. For small a > 0, 
the control on [0, a] with ui(t) = u is admissible for x. We let (£,w) be the 
corresponding trajectory. By continuity of \7V and of /(•, u), we may choose <r 
small enough so that 

*Wl = vvm)-f(xMt))<o 

for all t e [0,cr]. It follows that V(£(t)) < V{x) for all t > 0. I 

Observe that, for systems with no controls, condition (5.34) amounts to the 
requirement that LfV(x) < for all x € O, x ^ x Q ; this is the most classical case 
studied for differential equations. For continuous-time systems x = f(x) with no 
controls, it is customary to write just u V(x) v instead of (LfV)(x) = \7V(x)f(x). 

Example 5.7.5 Consider the system with X = R 2 , IX = R, and equations 

Xl = x 2 

x 2 = — sin^i + u 

where we take all notions with respect to the equilibrium state x° = (and 
corresponding input u° = 0). This corresponds to a model for a pendulum 
where we do not identify states whose angles differ by 27r. The squared norm 

V(x) := xl + x 2 2 

is a positive definite proper smooth function on R 2 , so we test to see if it is a 
local or global control-Lyapunov function for E. Using the criterion in Lemma 
5.7.4, we check the sign of 

W(x).f(x,u) = 2x\x 2 — 2x 2 sin^i + 2x 2 u . 

When x 2 ^ 0, this expression is indeed negative for an appropriate choice of 
u. But at any x such that x 2 = 0, it vanishes identically. Thus, the criterion 
cannot be applied. After some trial and error (typical of the use of the Lyapunov 
function technique), we try the alternative 

V(x) := 2xl + xl + 2x 1 x 2] 

which is still smooth, positive definite, and proper (since it is also a positive 
definite quadratic form). Now however, 

VV(x).f(x,u) = (4:X 1 +2x 2 )x 2 + (2xi + 2x 2 )(-smxi+u) . 

When xi + x 2 ^ 0, again the expression can be made negative by an appropriate 
choice of u. But the expression is automatically negative for any other nonzero 
x. Indeed, if x\ + x 2 = then the second term vanishes, but the first one is now 
—2x 2 , which is negative unless x 2 = 0, which can only happen if x\ = —x 2 = 
too. Thus, the expression is negative except at x = 0, and V is a global control- 
Lyapunov function for E. So E is globally asymptotically controllable. □ 
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For linear systems, one often considers quadratic Lyapunov functions. As 
usual, we write P > to indicate that a matrix P is symmetric and positive 
definite, P > to indicate positive semidefinite, and P < 0, P < for negative 
semidefinite and definite respectively. We remind the reader that, unless oth- 
erwise stated, asymptotic stability notions and Lyapunov functions for linear 
systems are always meant with respect to x° = 0. 

Lemma/Exercise 5.7.6 Let E be a linear (time-invariant) continuous-time 
system with no controls, x = Ax, and let P > 0, P G R™ x ™. Prove that the 
condition 

A'P + PA < 

is sufficient for V(x) := x'Px to be a Lyapunov function for E. □ 

Lemma/Exercise 5.7.7 Let E be a linear (time-invariant) discrete-time sys- 
tem with no controls over R, x + = Ax, and let P > 0, P € R™ x ™. Prove that 
the condition 

A' PA - P < 

is sufficient for V(x) := x'Px to be a Lyapunov function for E. □ 

Condition (5.34) in Lemma 5.7.4 is often too restrictive. The next condition 
is less restrictive, but applies only to systems with no controls. The strict 
inequality is replaced by a weak inequality, but an extra condition is imposed. 
The application of Theorem 17 to the type of function described in the next 
Lemma is a weak version of what is often called the LaSalle invariance principle. 

Lemma 5.7.8 Assume that E is a continuous-time system with no controls, 
that V : X — » R is a continuously differentiable map, and that the open set 
O C lis such that properties (1) and (2) in the definition of Lyapunov function 
hold. Then, V is a (local) Lyapunov function if the following properties hold: 

1. For each x G O, 

VV(x).f(x) < (5.35) 

2. whenever £ is a trajectory on the infinite interval X = [0, oo) for which 

W(£(t))./(£(t))=0 for alii, (5.36) 

necessarily £(£) = x° . 

Further, V is a global Lyapunov function provided that these conditions hold 
with O = X and V is proper. 

Proof. We start by picking an £o > so that property (1) in the definition of 
Lyapunov function holds, that is, {x € X | V(x) < £o} is compact and contained 
in O. Since V is continuous at x°, there is some S > such that V(x) < So 
for all x in the closed ball B$(x°). We claim that the definition of Lyapunov 
function is satisfied with O := B$(x°). 
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Observe that, since Bs(x°) C {x G X | V(x) < e } Q O, V is positive definite 
on O. Furthermore, V is proper, because the compact set {x G X | V(x) < e} 
is included in O for all e < Sq small enough (proof: otherwise, there is some 
sequence {xk}, contained in the compact set {x G X | V{x) < Sq} \B$(x°), such 
that V(xk) — ► 0; without loss of generality, Xk — > x, which gives a contradiction 
since by continuity of V it holds that V(x) = 0, but x £ Bs(x )). 

We need only to establish property (3). For this, take any x G O, x ^ x a , 
and consider the maximal solution £ with £(0) = x, which is defined on some 
interval of the form I = [0, a), with a < oo. 

We first show that a = 00. Condition (5.35) implies that V(^(t)) is non- 
increasing as long as £(£) remains in the original set O, for which it is in turn 
sufficient that it remain in the set 

S:={x\ V(x) < e } 

because of the choice of sq. Assume that there would be some T > in X so 
that V(£(T)) > £o- Since V(^(-)) is continuous, there is a first such T; thus, we 
may assume that V(£(£)) < £o for all t G [0,T). So V r (^(-)) is nonincreasing on 
[0,T], which implies that V(£(T)) < F(x) < £o ; a contradiction. We conclude 
that £ remains in the compact set S for all £ G I, so by Proposition C.3.6 in 
Appendix C, I = [0, oo), as desired. 

Thus, the trajectory is defined for all t, and V is nonincreasing. Assume 
that it were the case that 

V(£(t)) = V{x) for all t. 

It follows that dV/dt = identically, and hence condition (5.36) holds there. 
We conclude that £ = a; , contradicting the assumption that £(0) = x ^ x Q . 
Thus, there is some t so that V(£(£)) < V(x), and V is a Lyapunov function as 
desired. I 



Exercise 5.7.9 Use Lemma 5.7.8 to establish the following generalization of 
Lemma 5.7.6: Assume that E is as in Lemma 5.7.6, and P > 0, P G K nx ™. 
Prove that a sufficient condition for V(x) := x' Px to be a Lyapunov function 
for E is that 

A'P+iM < -C'C, 

(the inequality is understood in the usual sense for symmetric matrices: Q\ < 
Qi means that x'Q\x < x' Q^x for all x) where C is a p x n matrix (p some 
integer) for which the pair (A, C) is observable, that is, (A' , C") is controllable. 
(Hint: See Corollary 3.5.9 in Section 3.5.) □ 

Exercise 5.7.10 State and prove a discrete-time analogue of Exercise 5.7.9. □ 
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Example 5.7.11 Consider now a system with no controls corresponding to a 
damped harmonic oscillator (or a pendulum linearized about 8 = 8 = 0), 

±i = x 2 

&2 = — Xi - x 2 ■ 

We want to study its asymptotic stability (with respect to x° = 0) using Lya- 
punov function techniques. Again trying the squared norm V(x) := x\ + x 2 , 
this is now a global Lyapunov function. To check this, we apply Lemma 5.7.8. 
Note that 

VV(x).f(x) = -2x\ < 

for all x. Further, 

W(£(t))-/«(*)) = 

along a trajectory £ implies that £2 = along the trajectory, so also £2=0 and 
hence from the second equation £1 = 0. Thus, £ = 0, and the Lemma applies. 
Alternatively in this simple example, one can compute the eigenvalues of 

1 
-1 -1 

which have real part —^, so A is a Hurwitz matrix. □ 

Example 5.7.12 Consider the double integrator system (X = M 2 and U = R): 

Xi = x 2 
± 2 = u. 

The linear feedback law u = —x\ — x 2 globally stabilizes this system, cf. Ex- 
ample 5.7.11. We show now how to globally stabilize this system with a simple 
feedback law which is globally bounded in magnitude. Let 8 : R — > R be any 
locally Lipschitz and strictly increasing continuous function such that 8(0) = 
(for instance, we may pick 8(s) := tanhs, which is bounded). We show that the 
feedback law 

u := —8(xi + x 2 ) 

globally asymptotically stabilizes the given system. In order to construct a 
Lyapunov function, let 0(.s) := J Q 8(t) dt. This function is clearly positive 
definite and proper; we claim that 

V (£1,0:2) := ©(xi) + O(o:i + x 2 ) + x\ 

is a Lyapunov function for the resulting closed-loop system. Indeed, a simple 
calculation shows that 

VV(x)-f(x) = [8(x 1 )-8(x 1 + x 2 )]x 2 - [8( Xl +x 2 )} 2 . 

Since 8 is monotonic, the term [#(a;i) — 8(x\ + x 2 )\ x 2 is always nonpositive, and 
it is strictly negative unless x 2 = 0. If x 2 = 0, then the second term is negative, 
unless also X\ = 0. Thus, W (x) f (x) < for all x 7^ 0, as required. □ 
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Exercise 5.7.13 For the system in Example 5.7.11, find a quadratic global 
Lyapunov function V that satisfies the strict inequality condition in Lemma 
5.7.4, that is, VV{x).f{x) < for all x ^ 0. □ 

Exercise 5.7.14 Modify Example 5.7.11 to deal with the full nonlinear model 
of a damped pendulum with no controls, 

Xi = x 2 

±2 = — sinxi — X2 ■ 

Now all the states with xi = kir and X2 = are equilibria, so for x° = there is 
at most local asymptotic stability. From physical considerations, a reasonable 
Lyapunov function to try is 

V(x) = (1 -cosari) + (l/2)x2 



(sum of potential and kinetic energy), say for X\ € [— 7r/2, 7r/2], and 1 + (\j1)x\ 
for all other x\. Show that this is a local Lyapunov function. □ 

Exercise 5.7.15 Consider any system with X = R 2 , IX = R, and equations 

±i = /i(xi,aJ2) +ux[ 

X2 = f2{xi,X 2 ) +UX T 2 

where f\ and f^ are arbitrary and r € Z+. Then: 

1. Show, using a control-Lyapunov function, that if r is odd then this system 
is globally asymptotically controllable (with respect to the origin). 

2. Give examples to show that, when r is even, the system may or may not 
be locally asymptotically controllable. □ 

Exercise 5.7.16 Consider a continuous-time system x = f(x) with no controls 
and X = R™. Suppose that V : R" — » R is proper and positive definite, and 
satisfies V(x) = LfV(x) < for all x ^ (this is the Lyapunov condition in 
Lemma 5.7.4). Show that there exists a continuous function a : [0,oo) — ► [0,oo) 
which is positive definite (that is, a(0) = and a(r) > for all r > 0) such that 
the following differential inequality holds: 

W(cc) • f{x) = V{x) < -a{V{x)) for all x € R n . 

{Hint: Study the maximum of LfV(x) on the set where V(x) = r.) □ 

Exercise 5.7.17 O Suppose that V is as in Exercise 5.7.16. Show that there 
is a proper and positive definite W : R" — > R so that 

VW{x) ■ f{x) = W{x) < -W{x) for all x £ R™ , 

that is, provided we modify V , we can choose a = identity. [Hint: Try W = 
poV, for a suitably constructed p : [0,oo) — » [0,oo).) □ 
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Linear Systems 

For linear systems, Lyapunov functions can be obtained constructively; we now 
show that for any asymptotically controllable linear system there is always a 
global quadratic control- Lyapunov function. The material in Appendix C.5 on 
Stability of Linear Equations should be reviewed at this point, as well as Propo- 
sition A. 4. 6 and Corollary A. 4. 7 in Section A. 4 of the Linear Algebra Appendix. 

Since a continuous-time system E = (A, B) is asymptotically controllable 
if and only if there is an F such that x = (A + BF)x is asymptotically stable 
(Lemma 5.5.6), and similarly for discrete-time systems and convergent matrices, 
it is enough to prove the claim for systems without controls. 

We need first a very useful matrix Lemma. A matrix equation of the type 
MX + XN = Q is often called a Sylvester equation. 

Lemma 5.7.18 Let M and N be two fixed n x n matrices over K = 1 or C, 
and consider the linear operator 

C : K" x " -> K" x ™, C(X) := MX + XN . 

If both M and N are Hurwitz, then C is invertible. 

Proof. There are two proofs one can give: one algebraic and the other analytic. 
The algebraic proof uses Corollary A. 4. 7 in Appendix A. 4. This Corollary can 
be applied with A = M, B = —N'; since N and M are Hurwitz, M and —N' 
have no common eigenvalues. There is also a purely analytic proof, which results 
in an explicit formula for the solution, as follows. 

It will suffice to prove that C is onto. Thus, we need to show that given any 
matrix Q there exists some X so that 

MX + XN = Q . 

Since M and N are Hurwitz, 

||e tM Q e t7V ||<c||g||e 2At 

for all t > 0, where A < is any real number such that all eigenvalues of M and 
N have real parts less than A. Therefore, 



Jo 



P:= - I e tM Qe tN dt 
is well-defined. Further, 

/>CO 

MP + PN = - [Me tM Qe tN + e tM Qe tN N] dt 
Jo 



00 d{e tM Qe tN ) 
dt 

tM^tN 



dt 



Q- lim e tM Qe tN = Q. 

t— »oo 



as desired. 
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Theorem 18 Let A G K" x ™. The following statements are equivalent: 

1. A is Hurwitz. 

2. For each Q € K nx ™ , there is a unique solution P of the Lyapunov matrix 
equation 

A'P+PA = Q, 

and if Q < then P > 0. 

3. There is some P > such that A' P + PA < 0. 

4- There is some P > such that V(x) := x' Px is a Lyapunov function for 
the system x = Ax. 

Proof. Note that (2) trivially implies (3), and that (3) implies (4) (use Lemma 
5.7.4, or refer to Exercise 5.7.6). Also, (4) implies (1) by Theorem 17 (p. 220). 
(An alternative and direct proof that (3) implies (1) is also given in the proof 
of Lemma 8.4.7 in Chapter 8.) 

So it is only left to show that (1) implies (2). Pick any Q. Since A is 
Hurwitz, A' also is, and thus Lemma 5.7.18 shows that there is for each Q a 
unique solution P. The formula for P, 

P:= - f°° e tA ' Qe tA dt , 

shows that P > if Q < 0, since the latter implies that — e tA Qe tA is a positive 
definite matrix for each t. I 

Observe that, for example, when Q := —L the proof gives the Lyapunov 
function 



oc 



tA„ m2 



V(x) = / \\e tA x\\ dt. 
Jo 

Lemma/Exercise 5.7.19 Let A e R™ x ". The following statements are then 
equivalent: 

1. A is convergent (discrete-time Hurwitz). 

2. For each Q € K™ x ", there is a unique solution P of the discrete Lyapunov 
matrix equation 

A' PA -P = Q, 

and if Q < then P > 0. 

3. There is some P > such that A' PA - P < 0. 

4. There is some P > such that V(x) := x' Px is a Lyapunov function for 
the system x + = Ax. □ 
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Remark 5.7.20 As pointed out in Appendix C.5, there are direct algebraic 
tests for determining whether a matrix is Hurwitz, without computing eigenval- 
ues. The following is a simple example of one such test; it is more complex than 
those used in practice, but it is relatively trivial to establish. Given an n x n 
real matrix A, set up the equation 

A'X + XA= -I, 

and solve for symmetric X. This is a linear equation, so Gaussian elimination 
can be used. If no solution exists, then A is already known not to be Hurwitz. 
If a solution X is found, we check whether it is positive definite, for instance by 
evaluating all the principal minors of X. If X > 0, then A is Hurwitz, by part 
(3) of Theorem 18 (p. 231). If not, either the solution was not unique or the 
unique solution is not positive definite, so by part (2) of the Theorem we know 
that A is not Hurwitz. 

There is a more abstract way to prove the existence of an algebraic test for 
stability. Since the Hurwitz property can be expressed in first-order logic, one 
also knows from logical arguments based on the Tarski-Seidenberg Theorem for 
real-closed fields that Hurwitz matrices are precisely those whose entries satisfy 
some expression in terms of polynomial equalities and inequalities. □ 

Even though we have not treated stability of time-varying systems, it is 
worth pointing out through an example some of the difficulties that may arise 
in the more general theory. Consider a differential equation 

±{t) = A{t)x{t). 

It is a common mistake to assume that if A(to) is Hurwitz for each to then the 
differential equation must have some sort of stability property. The following 
exercise shows that this need not be the case. 

Exercise 5.7.21 For any fixed 1 < a < 2, consider the following matrix of 
functions of time: 



A(t) 



-l + o cos 2 t 1 — a sin t cos t 



■2 



- 1 — a sin t cos t — 1 + a sin t 
Prove that A(to) is Hurwitz for each fixed to G K, but that 

£(t) = e (°-D'( mst 
sv ; \-sint 

satisfies x(t) = A{t)x(t) but does not converge to the origin. □ 

Theorem 17 (p. 220) and Exercise 5.7.9 can be used to construct an explicit 
feedback matrix F as needed for part 4 of Proposition 5.5.6.: 
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Exercise 5.7.22 Suppose that the pair (A,B) is controllable. Pick any T > 
and consider the controllability Gramian (cf. Exercise 3.5.5) 

W := f e~ sA BB'e- sA ' ds . 
Jo 

Let F := —B'W~ l . Prove that A + BF is a Hurwitz matrix, i.e., the feedback 



a 



-B'W^x 



stabilizes x = Ax+Bu. (Hint: Prove, equivalently, that the transpose (A+BF)' 
is Hurwitz, by an application of Exercise 5.7.9, taking "P" there as W , "A" as 
{A+BF)', and "C" as B' . You will need to use that {A + BF, B) is controllable, 
and you will need to evaluate the integral appearing in the expression for AW + 
WA'.) U 

5.8 Linearization Principle for Stability 

Consider as an illustration the following discrete-time system with no controls 
and with X = K: 

x(t+l) = -x{t) + x 2 {t) . 
The linearization at x° = is the system 

x{t + l)= l -x{t), 

which is asymptotically stable. We may expect that the original system is itself 
(at least locally) asymptotically stable with respect to i° = 0. Indeed, the 
time-one mapping 

4>{t + 1, t, x) = V{x) = -x + x 2 

is a contraction on some interval [— e, e], because "P'(O) = 1/2 < 1. For instance, 
we may take e = 1/8. On this interval, x = is the only fixed point of V '. By 
the Contraction Mapping Theorem, it then follows that 

V n {x) -> 

for each initial 

xe [-1/8,1/8]. 

From this we may conclude that £ is locally asymptotically stable. Note by the 
way that E is not globally asymptotically stable, since x = 1/2 is an equilibrium 
point. 

We shall leave the general discrete-time case as an exercise and prove the 
continuous-time version of the above observation. 
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Theorem 19 Assume that E is a time-invariant continuous-time system of 
class C , 

x = f(x,u) 

with X C K" and IX C K m open, and fe£ (x , u ) &e an equilibrium pair, i.e., 

f(x°,u°) = 0. 

Assume that the linearization of E ai (x , u ) is asymptotically controllable. 
Then E is locally asymptotically controllable (to a ,). Moreover, there exists in 
that case a matrix F £ R mxrl g^c/j that the closed-loop system 

x = f cl (x) := f(x,u° + F(x - x )) (5.37) 

is locally asymptotically stable. 

Proof. It will be enough to prove the last statement, since this implies that E 
is locally asymptotically controllable. Let (A, B) be the linearization at (x a , u ). 
By Proposition 5.5.6, there is some F such that 

A d :=A+BF 

is Hurwitz. Pick the (unique) P > such that 

A' cl P + PA d = -I . 

Let E c ; be the system with no controls having the equation (5.37). 

We claim that the linearization of E c ; at x = x° is the linear system with 
matrix A c \. To establish this fact, write first 

f(x, u) = A(x - x°) + B{u - u°) +g{x-x a ,u- u°) , 

where 

||(a,/3)||-(> Jin. 



lim fe^# = 0. 



So it holds that 

fd(x) = A cl {x - x°) + j(x - x°) , 

where 

7 (x - x a ) = g(x - x°, F(x - x )) 

is of order o(x — x°), as wanted. 
Consider the function 

V(x) — {x-x^yPix-x ) 

on W 1 . For small enough e > 0, the sets 

{x£R n \ V{x) < e} 
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are included in X, so they are compact subsets of X. We take the restriction 
of V to X; thus (1) and (2) in the definition of Lyapunov function are satisfied 
for any open OCX. We now find an O so that the criterion in Lemma 5.7.4 is 
satisfied. Note that 

VV(x).f c i{x) = - \\x - x°\\ 2 + 2{x - x°)'Pj{x - x°) . 

Since 

||2(x-a; )'P7(a;-a; )|| ||7(x-o; )|| 

\\x-x°\\ 2 " C \\x-x°\\ ° 

as \\x — x \\ — » 0, it follows that \7V(x).f c i(x) < whenever \\x — x°\\ is small 
enough. This determines an open neighborhood of x° as desired. I 

Lemma/Exercise 5.8.1 Assume that £ is a time-invariant discrete-time sys- 
tem of class C , 

x + = V(x,u) , 

with X C K" and li C K m open, and let (x , U ) be an equilibrium pair, i.e. 

V(x°,u n ) = x°. 

Assume that the linearization of E at (a; ,u ) is asymptotically controllable. 
Then £ is locally asymptotically controllable (to a; ). Moreover, there exists in 
that case a matrix F G R mx ™ such that the closed- loop system 

x+ = Vd{x) := V(x, u° + F(x - x )) (5.38) 

is locally asymptotically stable. □ 

Exercise 5.8.2 O Our proof of the above Theorem is based on Lyapunov sta- 
bility ideas. An alternative proof could be based on a contraction argument as 
in the discussion at the beginning of this section, using that for linear systems 
with no controls the matrix A is Hurwitz (convergent in the discrete-time case) 
if and only if the map 

t A 

x i— > e x 

(or, x <— > A l x in discrete-time) is a contraction for t > large enough. Carry 
out such a proof in detail. □ 

Exercise 5.8.3 Use Exercise 5.7.11 and Theorem 19 (p. 234) to show that the 
nonlinear damped pendulum treated in Exercise 5.7.14 is locally asymptotically 
stable. □ 

Exercise 5.8.4 Find explicitly a matrix F G R lx2 as concluded in Theorem 

19 (p. 234), for the system (X = M 2 ,ll = E, x° = 0) 

±1 = X X X 2 + X 2 
X 2 = U. 

Show that £ is not globally asymptotically controllable. □ 
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First-Order Necessary Conditions for Stabilization 

Consider a continuous-time time-invariant system 

x = f(x,u) (5.39) 

of class C 1 and an equilibrium pair {x , u ). We say that this system is (lo- 
cally) C 1 stabilizable (respectively, smoothly stabilizable) with respect to this 
equilibrium pair if there exists a function 



k : X -> 11, k(x°) 



«° 



of class C 1 (respectively of class C°°) defined on some neighborhood Xo of x° 
for which the closed-loop system (with state-space Xo) 

x = f(x,k{x)) (5.40) 

is locally asymptotically stable. If Xo = X and (5.40) is globally asymptotically 
stable, we say that the system (5.39) is globally C 1 (or C°°) stabilizable. 

Theorem 19 showed that local smooth stabilizability holds provided that 
the linearization about the equilibrium pair is itself asymptotically controllable. 
This condition is not necessary, as illustrated by 

x = u , 

which has the linearization A = B = but which can be smoothly stabilized by 
k{x) := —x. However, there is a partial converse to the fact that stabilizability 
of the linearization implies local stabilizability. In order to present it, we first 
need a result which complements Theorem 18 (p. 231) on Lyapunov equations. 

Proposition 5.8.5 Assume that A € W ixn has at least one eigenvalue with 
positive real part. Then there exists a symmetric n x n matrix P, a constant 
c > 0, and a positive definite matrix Q such that the following two properties 

hold: 

1. A'P + PA = cP+Q and 

2. there is some x so that x' Px > . 

Proof. Assume that A has eigenvalues with positive real part. Let 

c := min{o | a > and a + ib is an eigenvalue of A} . 

Consider the matrix C := A — | J; this has no purely imaginary eigenvalues but 
still has some eigenvalues with positive real part. Decomposing into its stable 
and unstable parts, there exists an invertible matrix T such that 

TC7->=D=[' D J I 
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and both D\ and — D 2 are Hurwitz matrices. We will obtain a symmetric matrix 
P so that there is some x so that x' Px > and so that 

D'P + PD = I . (5.41) 

It will then follow that 

A'P + PA = cP + Q , 

with the choices P = T'PT and Q = T'T. Both P and Q are symmetric, 
and because T is invertible, it holds that Q > and there is some x so that 
x' Px > 0. Thus, it is only necessary to find P as above. 

Since D\ is Hurwitz, there exists a symmetric (positive definite) Pi of appro- 
priate size so that D[Pi + P\D\ = —I, by Theorem 18 (p. 231). Equivalently, 

D , 1 {-Px) + {-Px)D x =I. 

Similarly, since — D 2 is Hurwitz, there is a P 2 > so that (— D 2 )' P 2 -\-P 2 {— D 2 ) = 
—I, or equivalently 

D' 2 P 2 + P 2 D 2 = I . 

With 

\ P 2 

we conclude that (5.41) holds, and also (0 x)P I > for all x 7^ 0. I 

We wish to prove that certain systems are not locally (and hence not glob- 
ally) asymptotically stable. In order to establish this, we prove that the systems 
in question are not even "stable" in the sense of Lyapunov. We have not dis- 
cussed (non-asymptotic) stability, so we define directly the opposite concept: 

Definition 5.8.6 Let E be a time-invariant system with no controls, and let x° 
be an equilibrium state. The system E is unstable if there exists some e > so 
that, for each S > 0, there is some state x in the ball B$(x°) of radius 6 about 
x and there is some T € 7+ .so that 4>{T,x) is defined and <j)(T,x) (£ B £ (x°). □ 

Clearly, an unstable system cannot be asymptotically stable. 

Theorem 20 Let (E) : x = f(x) be a continuous-time time-invariant system 
with no controls, and let x° € X be an equilibrium state. Let V : B — > R be a 
C 1 function defined on the ball B = B £ (x°), and assume that there exists some 
constant c > such that 

W(x) := L f V(x) - cV(x) > for all xeB 

and so that 

x° Gclosja; | V(x) > 0} . (5.42) 

Then the system E is unstable. 
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Proof. We use the same e for the definition of instability. Pick any 5 > 0, 
assumed without loss of generality to be less than e. By the second assumption, 
there is some x 6 Bs(x°) for which V(x) > 0. Picking a smaller e if needed, 
we may assume that V is bounded by j3 on B. We need to show that it cannot 
happen that £ = tp(x) is defined and remains in B for all t > 0. Assume then 
that this is the case. 

Consider the function a(t) := e~ ct V(£(t)). This is differentiable, and its 
derivative satisfies 

a(t) = e- ct W{£{t)) > for all t > . 

Thus, e~ ct f3 > a(t) > a(0) = V(x) for all t, with c > 0, contradicting the fact 
that V{x) > 0. I 

Corollary 5.8.7 Let (£) : x = f(x) be a continuous-time time-invariant sys- 
tem with no controls, and let x° € X be an equilibrium state. Let A be the 
linearization matrix at x° , that is, 

x = A(x - x°) + g(x - x°) , 

where g(x) is a continuous function that has order o(x) as x — > 0. If A has any 
eigenvalues with positive real part, then E is unstable. 

Proof. Let P, Q, c be given as in Proposition 5.8.5, and let V(x) := (x — 
x )' P(x — x Q ) in Theorem 20, restricted to X. Note that, if y' ' Py > then 
V{x° + Xy) > for all A ^ 0, so (5.42) holds. 
With the same c, 

W{x) = L f V{x) - cV{x) = (x- x°)'Q(x - x°) + 2{x - x a )'Pg{x - x°) . 

If we prove that W(x) > for all small enough x, the result will follow. For 
this, note that 

x 1 ' Qx 



, > a for each x ^ . 
Nil 2 " 
where a is the smallest eigenvalue of Q, while 

\2x'Pg(x)\ \\g(x)\\ 



Nf 
which is smaller than a for all small x. I 

Corollary 5.8.8 Let £ be a continuous-time time-invariant system of class C . 
Let (x° , u°) be an equilibrium pair for S, and let (A, B) be the linearization at 
this pair. Assume that (A, B) has an uncontrollable eigenvalue with positive 
real part, that is, that x (Definition 3.3.5) has a root with real part > 0. Then 
the system E is not locally C 1 stabilizable. 
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Proof. Assume that the system would be C 1 stabilizable by means of the 
feedback law k : X — » U, and write 

k{x) = F(x - x°) + g 2 (x - x a ) , 

where g 2 {x) = o(x) as x — > 0. Note that A + BF still has an eigenvalue with 
positive real part, by (the straightforward part of) Theorem 13 (p. 186). If we 
also expand 

f{x, u) = A(x - x°) + B{x - x") + gi {x - x°) 

with gi(x) = o(x), we may conclude that the closed-loop equations are 

x={A+ BF){x - x°) + g{x - x°) , 

where g(x) := Bg 2 (x) + g\{x, k{x)) is again of order o(x). But the closed-loop 
system cannot be stable, by Corollary 5.8.7. I 

Since the zeros of x are the right-hand plane numbers for which the rank 
of [si — A, B] drops (by Lemma 3.3.10), we may summarize the discussion as 
follows, for time- invariant continuous-time systems E and their linearizations: 





rank [si — A, B] = n for all Re s > => E locally C 1 stabilizable 










rank [si — A, B] < n for some Re s > => E not locally C 1 stabilizable 



The critical case, when the only uncontrollable unstable eigenvalues are 
purely imaginary, gives rise to the more interesting theory, since the linearization 
does not characterize stabilizability. 

It is important to emphasize that the above results refer to C 1 stabilizability. 
There may exist, even in the second case, a feedback that is smooth everywhere 
but at x°, and even continuous everywhere, which gives an asymptotically stable 
system. An example of this is provided by the system 



u 3 



(x° = and u° = 0) for which the feedback u := —2a; 1 ' 3 stabilizes. Proposi- 
tion 5.9.10 below deals with stabilization by k(x) smooth away from the origin. 

5.9 Introduction to Nonlinear Stabilization* 

The previous section discussed first-order conditions for stabilization. One of the 
conclusions was that, if the linearization about the equilibrium being considered 
has any uncontrollable eigenvalues with positive real part, then C 1 stabilizability 
is impossible. Thus, from the point of view of the theory of local C 1 stabilization, 
the "critical case," in which there are some purely imaginary uncontrollable 
eigenvalues, is the one to be studied further. 



* This section can be skipped with no loss of continuity. 
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In this section, we discuss several further issues regarding nonlinear stabi- 
lization. We start with three different approaches to building feedback. All are 
Lyapunov-function based, and the resulting techniques have found wide appli- 
cability, both for local (in critical cases) and global stabilization. The first two, 
"damping control" and "backstepping" , allow the construction of feedback that 
is smooth at the origin, while the third one, the use of "universal formulas" for 
stabilization, typically guarantees smoothness only away from the equilibrium 
of interest. 

We close the section with "Brockett's necessary condition" for nonlinear 
stabilization. 

Damping Control 

The first method starts with a V which insures boundedness of trajectories 
(typically, V is an energy function associated to a conservative system), and 
adds a "friction" term implemented through a feedback law. 

Proposition 5.9.1 We consider smooth control- affine systems 

rn 

x = f(x) + ^ j u l g i (x) 

as in Equation (2.34) (but writing here / instead of <7o) with X = R™, IX = K m , 
and /(0) = 0. Suppose that V is smooth, positive definite, and (globally) 
proper, and satisfies 

L f V{x) = VV(x) ■ f{x) < for all x £ X. (5.43) 

Consider the following set: 

Q:= { x \ L f V(x) = and L k f L gi V(x) = for all k = 0, 1, 2, . . . , i = 1, . . . ,ro} 

and suppose that Q = {0}. Then, the following smooth feedback law: 

u = k(x) := - (VV(x) ■ G(x))' = -(L gi V(x),...,L gm V(x))' (5.44) 

globally asymptotically stabilizes the system (with respect to x° = 0, u — 0). 

Proof. This is a simple application of the LaSalle invariance principle, cf. 
Lemma 5.7.8. We consider the closed-loop system 



F(x) = f{x)-Y J L a% V{x)g l {x) 



and note that 

m 

VV{x).F{x) = L t V{x)-Y t {L gi V{x)f 
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is nonpositive for all x, by assumption. Take now any trajectory £ for which 
VV(£(t)).F (£(£)) = 0; we need to show that £ = 0. Along such a trajectory, it 
must be the case that L/V(£(£)) = and also L gi V(£(t)) = for every i. For 
any given i, also the time derivatives of L gi V(£(t)) must vanish identically. The 
first derivative of this function is LfL g .V(£(t)) (chain rule), and, inductively, 
the fcth derivative is LhL g .V(£(t)). It follows that £(£) G Q for all t, so indeed 

£ = o. ' I 

Exercise 5.9.2 This problem is a variation of Proposition 5.9.1. Suppose given 
a smooth control-affine system, and V as there, so that (5.43) holds, and con- 
sider, for each iGl, the following vector subspace of R" 



A(x) := span < f(x), ad^i(x), i = 1, . . . ,m, k 



0,1,2,. 



(in the differential-geometric terminology of Chapter 4, A defines a "distribu- 
tion"). Assume that \7V(x) = implies x = and that 

dim A(x) = n for all x ^ . 

Show that the feedback law (5.44) stabilizes the system. (Hint: Prove by 
induction on k that, if £ is a solution of x = f(x) so that LfV(£(t)) = and 
L gi V($(t)) = for all i, then also L adkg .V (£(£)) = 0, for all k and all i You will 
need to use the facts that L ad k+i can be expressed in terms of LfL ad k and 
L ad k Lf, and that V (LtV) vanishes on the set {x \ LrV(x) = 0} (why?).) □ 

Exercise 5.9.3 Give a feedback law u = k(x), fc(0) = 0, that stabilizes the 
bilinear system 

-i oJ+Hi o 

(Use the above results.) □ 

Exercise 5.9.4 O As in Exercise 4.3.15, consider the problem of controlling the 
angular momentum of a rigid body by the use of one or more applied torques. 
For concreteness, we assume that three torques are being applied. Thus (cf. 
Exercise 4.3.15), we have a system with X = K 3 , U = K 3 , and equations given 
by: 

Ix = S{x)Ix + Tu, (5.45) 

where here / denotes the diagonal matrix with entries I\, I21I3, T € R 3x3 , and 



S(x) 



It is assumed that Ii 7^ Ij whenever i 7^ j. We may also write this system in the 
more standard form x = f(x) + G(x)u, where G(x) = (gi (x), 52 (x), 33(2;)) = 






x 3 


-x 2 


~%3 





Xl 


X2 


-x\ 
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I~ 1 T is constant and f(x) = I~ 1 S(x)Ix. Motivated by the above results, we 
consider the following function (the kinetic energy) : 

V{x) := - (hx\ + I 2 x\ + I 3 xj) 

and, associated to it, the following feedback law: 

k(x) = - (VV(i) • G(x))' = -(L gi V(x),...,L gm V(x))' = -T'x. 

Prove that this feedback stabilizes the system if and only if every row of T is 
nonzero. (Hint: You need to prove that Q = {0}. For this, you will need to 
consider the various cases rankT = 1,2,3 separately. The computations in each 
case are geometric, using the fact that Q may be written as intersections of 
appropriate lines, cones, and ellipsoids.) □ 

Backstepping 

The following Lemma is useful in establishing that, under certain conditions, 
a series connection of two systems is stabilizable provided that each system is. 
For simplicity, we give the smooth version, but it will be clear from the proof 
how to relax to less differentiability. This result, often called the "integrator 
backstepping lemma," allows one to recursively design controllers for certain 
complex systems by a step-by-step procedure, starting with a simpler system 
and adding at each stage an integrator "in front" (at the input) of the simpler 
system. 

Lemma 5.9.5 Assume that the smooth system (with X open in K™ and IX open 

in R m ) 

x = f(x,u) 

is smoothly locally stabilizable using u = k(x) and that there is a local Lyapunov 
function V as in Lemma 5.7.4 for the system (5.40), which is smooth. Let 

h : X x It -> R m 

be smooth, h(x , u ) = 0. Then, the system 

x = f(x,z) (5.46) 

z = h(x,z) + u (5-47) 

having control set K m is again smoothly locally stabilizable (with respect to the 
equilibrium point (x°,u°) and the control 0). The same result holds if "global" 
is used throughout. 

In particular, the system obtained by integrating controls (h = 0) is again 
stabilizable. 
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Proof. Assume that (5.40) is asymptotically stable and that 

VV(x).f{x,k(x)) <0 



(5.48) 



for x t^ x° . 

For any state on the surface S = {(x,z) \ z = k(x)} such that x ^ x Q , 
W{x).f{x, z) < 0; thus, x moves towards a; , or, more precisely,!^ (x) decreases, 
as long as the trajectory stays in S. See Figure 5.6 (a; = and u° = there). 
The idea of the proof will be to construct a controller which forces every state 
that is not on S to approach S. The dashed lines in Figure 5.6 indicate the 
desired motions. One obvious way to force moves towards S is to find controls 
so that || z — A;(a;)|| is decreased, in addition to V(x) being decreased. Thus, we 
consider the following (smooth) function: 

W(x,z) := V(x) + i||z-fc(a;)|| 2 

as a candidate control-Lyapunov function. Note that W(x, z) will be small if 
both z w k(x) and x ~ x°, which implies z ~ u . 



z=k(x) 



K- z * > ^ 



lk.-*M 




Figure 5.6: Desired surface z = k{x). 

The derivative of W , along a trajectory corresponding to a control u, is: 

VV(x).f(x,z) + (z-k(x))'(h(x,z)+u-k*[x]f(x,z)). (5.49) 

Let us write 

u = — h(x,z) + k*[x]f(x, z) + u, 
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where it is to be chosen as a function of x and z. Then (5.49) becomes 

VV{x).f(x,z) + {z-k{x))'u 

and we are left with trying to make this expression negative. 

Since / is smooth, there is an n x m smooth matrix G such that 

f(x, k(x) + z) = f(x, k{x)) + G(x, z)z 

for all (x,z), or equivalently, 

f(x, z) = f{x, k{x)) + G(x, z - k(x))(z - k{x)) (5.50) 

for all (x,z). (For instance, if we write a(A) := f(x,k(x) + Xz), then from 
a (I) = a(0) + L a' (A) dX we conclude that one choice is 

G(x, z) := / f z (x, k(x) + Xz) dX , 
Jo 

where f z is the Jacobian of / with respect to z.) Since already \7V(x).f(x, k(x)) 
is negative, for x ^ X , and since now z—k(x) appears as a factor, we may simply 
use u = —[\7V(x).G(x, z — k(x))]' + u to first subtract the effect of G, and then 
stabilize using an appropriate u, for instance — (z — k{x)). Formally, then, we 
consider the following feedback law: 

k(x, z) := —h(x, z) + k*[x]f(x, z) — [\7V(x).G(x, z — k(x))] — z + k(x) . 

Note that k{x°,u°) = and that k is smooth. Along the trajectories of (5.46)- 
(5.47) with this feedback law, the derivative of W is 

\7V{x).f{x, z) + \h(x, z) + k{x, z) - fc* [x]f(x, z)] (z - k{x)) , 
which, by (5.50), reduces simply to 

VV(x).f(x,k(x))-\\z-k(x)f . 

This latter expression is negative whenever (x, z) ^ [x , u ). Thus, W is a Lya- 
punov function for the composite system, which is then asymptotically stable. 
The arguments can be done either locally or globally. I 

Example 5.9.6 Consider the system E with X = IX = R and 

x = u 3 . 

This system is (globally) stabilizable (relative to x° = 0) using the feedback 
u = k(x) := —x. The Lyapunov function V(x) = x 2 can be used. We wish to 
show that the new system £' with equations 

x = z 3 

z = u 



5.9. Introduction to Nonlinear Stabilization* 245 

is again smoothly stabilizable (relative to (0,0)). We have the expansion 

f(x,k(x)+z) = {k{x) + zf 

= {k{xjf + [3{k{x)) 2 + 3k(x)z + z 2 ]z 
= f(x,k(x)) + G(x,z)z 1 

where G(x, z) = 3a; 2 — "ixz + z 2 . Equivalently, 

f(x,z) = f(x,k(x)) + G(x,z — k(x))(z — k(x)) 
= f(x, k(x)) + (z 2 — zx + x 2 )(z + x) . 

In the above proof, we obtain 

W(x,z): = V(x) + hz-k{x)) 2 
= x 2 + -(z + x) 2 

and the feedback law 

k(x, z) = — z — [2x(z — zx + x )} — z — x . 

Note that fc(0, 0) = and that k is a polynomial. Along the trajectories of £' 
with this feedback law, the derivative of W is 

W = [2x + (z + x)]z 3 + (z + x) [-z 3 - 2x(x 2 - xz + z 2 ) - x - z] 
= -2x i -{z + x) 2 , 

which is negative whenever (x,z) ^ (0,0). This proves that IF is a Lyapunov 
function for the composite system £', which is therefore (globally) asymptoti- 
cally stable. □ 

Exercise 5.9.7 Consider the following system, with X = M 3 and IX = R: 



Xl 


= 


x 2 


X2 


= 


X3 


x 3 


= 


u 



(and x° = 0, u° = 0). Find a smooth globally stabilizing feedback law. □ 

Exercise 5.9.8 The equations for the angular momentum of a rigid body con- 
trolled by two independent torques (e.g., a satellite controlled by two pairs of 
opposing jets, cf. also Exercises 4.3.15 and 5.9.4) can be written, for appropriate 
parameter values and after some simplification, as: 

±1 = CI1X2X3 + Ul 

x 2 = a 2 Xix 3 + u 2 

X 3 = CI3X1X2 
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where oi, ai, and 03 are certain constants and a^ ^ 0. We view this as a system 
with X = K 3 and IX = K 2 . Prove that this system can be globally smoothly 
stabilized (about x = and u = 0), and exhibit a smooth stabilizing feedback. 
(Hint: Apply Lemma 5.9.5.) □ 

A Universal Formula for Feedback 

Several approaches to modern feedback design start with a control-Lyapunov 
function for a system, and use the control-Lyapunov function in order to con- 
struct a feedback stabilizer. We now describe one technique which has proven 
useful in that regard, restricting attention to control-affine continuous-time 
time-invariant systems 

x = f(x) + uigi(x) + ... + u m g m {x) = f{x) + G{x)u (5.51) 

having X = 1" and II = K m (as in Equations (4.18), (4.22), and (5.10), except 
that, here, we prefer to write / instead of gg). We suppose that the system 
is smooth, and that /(0) = 0, and discuss stabilization with respect to the 
equilibrium x = 0, u Q = 0. 

Given is a smooth global control-Lyapunov function V with the infinitesimal 
decrease property which was described in Lemma 5.7.4. That is, V : K™ — » R is 
proper and positive definite, and, for each nonzero x G E™, there is some u G K m 
so that Equation (5.34) holds. When specializing to control-affine systems, this 
condition says that, for each i/O, there must be some u G K m so that 

LfV{x) + Ul L gi V{x) + . . . + u m L gm V{x) < (5.52) 

or, equivalently, 

L gi V(x) = ... = L gm V(x) = and x^ => L f V(x)<0. (5.53) 

Given such a V, one obvious way to try to control the system to zero is to use 
a feedback law u = k{x) which selects, for each x 7^ 0, some u so that (5.52) 
holds. The value of V will then decrease along trajectories, and properness 
and positive definiteness will assure that the trajectory itself converges to zero. 
There are many ways to select such an u as a function of x. For example, if we 
can be assured that a suitable u can be found in a compact subset Uq C K m (Uo 
may be allowed to depend on x, if desired), and if there is a unique minimizer 
to 

min LfV(x) + u\L gi V(x) + . . . + u m L gm V(x) 

u£Uo 

then one could pick u = k(x) to be this minimizer. Such an approach is a 
useful one, and it appears often in applications. It might give rise, however, to 
technical difficulties. For example, the function u = k[x) may not be sufficiently 
regular so that the closed-loop differential equation has unique solutions for each 
initial condition. Also, the minimization may not be trivial to perform, from a 
computational point of view. In addition, smoothness of k(x) is often desirable, 
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for implementation reasons. We show next that it is always possible to select 
a u = k{x) that is a smooth function of x, at least for x 7^ 0. (If we would be 
content with a class C l feedback k, with £ < 00, it would be sufficient to assume 
that V, as well as the system, are merely of class C e+1 .) That is, we look for 
a mapping k : R™ — » M. m , with k(0) = and so that k restricted to R" \ is 
smooth, so that 

VV(x).[f(x) + k 1 (x)g 1 (x) + ... + k m (x)g m (x)} < (5.54) 

for all nonzero x. Note that, if any such k happens to be continuous at the 
origin, then the following property, the small control property for V, must hold 
as well (take u := k(x)): 

For each e > there is a S > so that, whenever < ||x|| < 8, there 
is some u with \\u\\ < e such that (5.52) holds. 

Exercise 5.9.9 Suppose that F : R™ — > R™ is locally Lipschitz when restricted 
to R™ \ {0} and satisfies F(0) = 0, and assume that V is a proper and positive 
definite continuously differentiable function such that \7V(x) ■ F(x) < for all 
nonzero x. (For example, F = f + J^ kigi, where k, with fc(0) = 0, is smooth on 
K™ \ {0} and (5.54) holds for all nonzero states.) Show that there is a unique 
solution of x = F(x), for each initial condition, defined for all t > 0. Moreover, 
every solution converges to zero, and for each e > there is some S > so that 
||x(0)|| < S implies that ||x(i)|| < e for all t > 0. That is, the system x = F(x) 
is globally asymptotically stable. As an illustration, compute the solutions, for 
each initial condition and each t > 0, of the one-dimensional system x = — x^ 
{V(x) = x 2 is as required for this F). (Hint: On the open set X := R" \ {0}, 
there is local existence and uniqueness, and completeness can be proved using 
properness of V . Properness and positive definiteness also guarantee that is 
an equilibrium, so behavior such as that of x = +xs cannot occur.) □ 

Proposition 5.9.10 Let the smooth and positive definite V : R" — > R be so 
that, for each x ^ 0, there is an u € M m so that (5.52) holds. Then there exists 
a k, k(0) = 0, k smooth on R n \ 0, satisfying (5.54). Moreover, if V satisfies the 
small control property, then k can be chosen to also be continuous at 0. 

Proof. We shall prove the theorem by constructing a fixed real-analytic function 
if of two variables, and then designing the feedback law in closed-form by the 
evaluation of this function at a point determined by the directional derivatives 
VV(x).f(x) and W(a:).0i(x)'s. 

Consider the following open subset of M. 2 : 

S := {(a, b) eR 2 I b> or a < 0} . 

Pick any real analytic function q : R — > M. such that q(0) = and bq(b) > 
whenever b 7^ 0. (Later we specialize to the particular case q(b) = b.) We now 
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show that the function defined by: 



ip{a,b) :■■ 



if 6^0 



if b = 

is real-analytic on S. For this purpose, consider the following algebraic equation 
on p: 

F(a,b,p) = bp 2 - 2ap-q(b) = (5.55) 

which is solved by p = <p(a, b) for each (a, b) £ S. We show that the derivative 
of F with respect to p is nonzero at each point of the form (a, 6, <p(a, b)) with 
(a, b) £ S, from which it will follow by the Implicit Function Theorem that ip 
must be real- analytic: 

IdF , 

o~^~ = bp-a 

2 op 

equals —a > when 6 = and \J a 2 + bq(b) > when 6^0. 
Assume now that V satisfies (5.52). We write 

a(x) := W{x).f{x), bi(x) := W(x).gi(x), i = l,...,m, 

and 

m 

B(x) := (6i(ar), . . . , b m (x)), (3(x) := \\B(x)\\ 2 = ^ b 2 (x) . 

i=i 

With these notations, condition (5.53) is equivalent to: 

(a(x), /3(x)) £ S for each x ^ . 

Finally, we define the feedback law k = (ki, . . . , k m )' , whose coordinates are: 

ki(x) := -bi(x)<p(a(x),f3(x)) (5.56) 

for x 7^ and fc(0) := 0. This is smooth as a function of x ^ 0. Moreover, at 
each nonzero cc we have that (5.54) is true, because the expression there equals 

a{x)-<p{a{x),p{x))[3{x) = - ^J a{ X y + 0(x)q{0{x)) < 0. 

To conclude, we assume that V satisfies the small control property and 
show that, with the choice q(b) := b, the obtained function k is continuous at 
the origin. Pick any e > 0. We will find a 5 > so that ||fc(x)|| < e whenever 
||x|| < 5. Since k(x) = whenever 0(x) = 0, we may assume that 0(x) ^ in 
what follows. Let e' := e/3. 

Since V is positive definite, it has a minimum at 0, so W(0) = 0. Since the 
gradient is continuous, every bi(x) is small when x is small. Together with the 
small control property, this means that there is some 5 > such that, if x ^ 
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satisfies ||x|| < 5, then there is some u with ||w|| < e' so that a(x) + B(x)u < 
and 

\\B(x)\\<e'. (5.57) 

Since a(x) < —B(x)u, the Cauchy-Schwartz inequality gives a(x) < e'||B(a;)||. 
Thus 

|o(a:)|< e , ||B(a:)|| (5.58) 

provided < ||a;|| < 6 and a(x) > 0. On the other hand, observe that 



b(f(a,b) = a + \J a 2 + b 2 < 2 \a\ + b 

for every (a, b) € S for which b > 0. Thus, if < ||x|| < S and a(x) > 0, 
necessarily 

and hence also, using (5.57), 

||fc(aO|| = tp(a(x),P(x)) \\B{x)\\ < 3e' = e 
as desired. There remains to consider those x for which a(x) < 0. In that case, 

< a(x) + y / a{x) 2 + (3(x) 2 < j3(x) 
so < ip(a(x), f3(x)) < 1 and therefore 

\\k(x)\\=<p(a(x),P(x))\\B(x)\\ <e'<£ 
as desired too. I 

It is worth writing out formula (5.56) explicitly in the special case m = 1: 



k(x) 



L f V(x) + ^\L f V(x)] 2 + \L g V(x)f . tT „,^ n 

L^V{x) *L g V{z)*0 

ifL fl ^(x) = 0. 



As an example, consider the case of one-dimensional systems with a scalar con- 
trol (to = n = 1), 

X = f{x) + ug(x) . 

This system is stabilizable if the following assumption holds: if g{x) = and 
x ^ then xf(x) < 0. The feedback law given by the above construction, using 
the Lyapunov function V{x) = x 2 /2, is simply 



= xf(x) + \x\y/f(x)* + x*g(x)4 
xg(x) 

(which is smooth, even though the absolute value sign appears, because in the 
one dimensional case there are two connected components of M. — {0},) so that 
the closed-loop system becomes 

x = —sign (x)\/f(x) 2 + x 2 g(x) 4 . 
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Exercise 5.9.11 The function tp can be obtained as the solution of an opti- 
mization problem. For each fixed x ^ 0, thought of as a parameter, not as a 
state, we may consider the pair (a(x), B(x)) as a (l,m) pair describing a linear 
system of dimension 1, with m inputs. The equations for this system are as 
follows, where we prefer to use "z" to denote its state, so as not to confuse with 
x, which is now a fixed element of R n \ {0}: 

771 

z = az + y_. i>i u i = az + Bu . (5.59) 

The control-Lyapunov function condition guarantees that this system is asymp- 
totically controllable. In fact, the condition "£> = => a < 0" means pre- 
cisely that this system is asymptotically controllable. A stabilizing feedback 
law k = k{x) for the original nonlinear system must have, for each fixed x, the 
property that 

m 

a + V" bik t < . 

This means that u = kz must be a stabilizing feedback for the linear sys- 
tem (5.59). Consider for this system the infinite- horizon linear-quadratic prob- 
lem of minimizing (cf. Theorem 41 (p. 384)) 

oo 

u 2 {t) + (3{x)z 2 {t)dt. 
o 

(For motivation, observe that the term u 2 has greater relative weight when f3 is 

small, making controls small if x is small.) 

Prove that solving this optimization problem leads to our formula (5.56). □ 

We next turn to other necessary conditions for stabilizability. As we will 
remark, both of these hold even for weaker notions than C 1 stabilizability. 

Topological Obstructions to Stability 

When dealing with time-invariant systems with no controls, we write just <f>(t, x) 
instead of cj)(t, 0, x, u>) . 

Let S be any (topological) system with no controls and x° an equilibrium 
state. Assume that the system is locally asymptotically stable (with respect to 
x°). The domain of attraction of x° is the set 

D = D{x°) :={xeX\ lim cf>(t,x) = x } . 

t — >oo 

We need the following Lemma: 

Lemma 5.9.12 For each x G D and each neighborhood V of a; there exists 
some neighborhood V' of X, V' C D, and some T G T + , such that for all y € V 
it holds that 

<j)(t,y)€V for all t>T. 

In particular, D is open. 
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Proof. Let W be an open neighborhood of x° included in the domain of at- 
traction D and for which all trajectories starting in W remain in V; such a 
neighborhood exists by the definition of asymptotic stability. 

Now pick any point x in D. By definition of D, there exists some T > 
such that 4>{T,x) G W. By continuity of 4>(T, ■), there is then a neighborhood 
V' of x such that 

(j)(T, y) e W for all y E V . 

Since trajectories starting at 4>(T,y) are in D, it follows that V' C D. Since 
these stay in V, the other conclusion follows too. I 

Theorem 21 IfT = K and 4>(t,x) is continuous on t (as happens with contin- 
uous-time systems), then the domain of attraction is a contractible set. 

Proof. To prove contractibility we need to provide a continuous mapping 

H : [0,1] x D-> D 

such that H(0,x) = x for all x E D and (for instance) H(l,x) = x° for all x. 
We define H as follows, for t < 1: 

H(t,x):=4>(^- t , 

and H{1, •) = x° . By Remark 3.7.3, H is continuous in both variables, for t < 1. 
We are left to prove only continuity at each (l,a;). Take any such x, and 
pick any neighborhood V of H(l,x) = x . We wish to find some 5 > and 
some neighborhood V' of x such that 

H (s, j/)£ V whenever y G V' and 5 > 1 — 5 . 

By Lemma 5.9.12, there is a V' and a T so that this happens provided that 
Y^-j > T. It is only necessary then to pick S := yxy. I 

Corollary 5.9.13 If E is a continuous-time system for which X is not con- 
tractible, then £ is not C 1 globally stabilizable (about any x°). 

Proof. If the system were stabilizable, the resulting closed-loop system would 
be a continuous-time system that is asymptotically stable and for which the 
domain of attraction is all of X. I 

Example 5.9.14 Consider the continuous-time system on X = K 2 — {0} and 

IX = E 2 , 

X\ = U\ 

x 2 = u 2 ■ 



p = 


PUl 


6 = 


u 2 . 


( Ul 


-u 2 


\u 2 


Ul 
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This system is completely controllable, because any two points in the plane 
can be joined by a differentiable curve that does not cross the origin and any 
such curve is a path for the system. Furthermore, every point is an equilibrium 
point. About any x° there is local smooth stabilizability since the linearization 
has the same equations as the original system, but considered as a system with 
X = R 2 . More explicitly, we may use the control law Ui := — (xi — x®), which 
is well-defined on some neighborhood of any (nonzero) a; . But, for no a; is the 
system globally smoothly stabilizable, because X is not a contractible space. 

This system is not complete (for instance, the control constantly equal to 
(1, 0)' on [0, 1] is not admissible for x = (—1, 0)'); a similar but complete example 
is as follows: Take the same state space and equations 



in polar coordinates or 



in the (xi,x 2 ) coordinates. □ 

Remark 5.9.15 The result in Corollary 5.9.13 can be extended to show that 
there does not exist for noncontractible X a globally stabilizing feedback k for 
which the closed-loop system is just locally Lipschitz on X. Indeed, for any such 
k we may consider the closed-loop system defined by the equation 

x= f{x,k(xj) . 

By the local existence Theorem, and the well-posedness Theorem 55 (p. 486) in 
the Appendix on differential equations, this defines a topological system, and it 
also satisfies the hypotheses of Theorem 21. □ 

Brockett's Necessary Condition 

There is a powerful and easy to check necessary condition for local stabilization, 
which we present next. As a motivation, consider first the very special case of 
a continuous-time system x = f(x, u), of dimension one, with X being an open 
subset of K which contains zero. If there would be a feedback law u = k(x) which 
stabilizes this system with respect to x° = 0, then clearly it must be the case that 
f(xi, k(xi)) > for some (actually, for all) x\ < 0, and that f(x 2 , k{x 2 )) < for 
some x 2 > (otherwise, if, for example, f(x, k(x)) > for all x near zero, then 
no trajectories £ of the closed-loop system with initial condition £(0) > can 
approach 0). It follows, using continuity of x i— » f(x, k(x)) and connectedness of 
[ari, X2] (i-e., the Intermediate Value Theorem), that there is some open interval 
V, containing zero, with the following property: for each p € V, there is some 
x £ X and some u £ 11 (namely, u = k{x)) so that f(x,u) = p. In other words, 
(x,u) <— > f(x,u) contains a neighborhood of zero in its image. For arbitrary 
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dimensions but linear systems, the same statement is true. Indeed, asymptotic 
controllability (with respect to x° = and u° = 0) implies, via the Hautus 
condition: 

rank [A, B] = n , 

which means that the mapping (x, u) <— > Ax + Bu must be onto (since this is a 
linear map, the conclusion is equivalent to asking that there exists a neighbor- 
hood of zero included in its image). There is a common generalization of these 
two observations: 

Theorem 22 Assume that the C 1 continuous-time system E: 

x = f(x,u) 

is locally C 1 stabilizable with respect to x . Then the image of the map 

/:XxU^I" 

contains some neighborhood of X . 

We prove this Theorem below, after reviewing a result from nonlinear anal- 
ysis. First, we illustrate with an application. 

Example 5.9.16 Consider the system on 1 = R 3 and IX = K 2 with equations: 



Xl 


- 


Wl 




±2 


= 


«2 




X 3 


= 


X 2 Ul - 


- x x u 2 



and equilibrium state 0. No point of the form 

( £ /0) 



is in the image of /, so there is no C 1 feedback stabilizing the system. This 
system, incidentally, is completely controllable, a fact that is easy to establish 
using Lie-algebraic techniques. It is also a bilinear system. Note that the 
linearized system has a single uncontrollable mode, A = 0. (This is usually 
called "Brockett's example"; see also Exercises 4.3.14, 4.3.16, and 9.4.2, and 
Example 9.2.14.) □ 

The result from nonlinear analysis that we need is as follows; it depends on 
basic facts from degree theory which are cited in its proof. 

Fact 5.9.17 Let B p (0) be the closed ball of radius p centered at the origin in 
R™ and S p its boundary: 

S P = {x\ \\x\\=p}. 
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Assume given a continuous function 

H : [0, 1] x B p {0) -» R" 

such that H(l,x) = —x for all x and 

H(t,x) ^ 

for all x £ S p and all t. Let 

F:B p (0) ^R n : x^H(0,x). 

Then there exists some e > such that the image of F contains the ball B e (0). 

Proof. For any continuous function G : B p (0) — » R™ and any p G B p (0) for 
which G(x) ^ p for all x € S p , one defines in nonlinear analysis the degree of G 
with respect to p, deg(G,p). (If G is C 1 and if detG*[#] 7^ for every x in the 
preimage G^ 1 (p) - which is a finite set, by the Implicit Mapping Theorem and 
the compactness of B p (0) - then deg(G,p) is Y1 x ^g- 1 (v) s ^S n (detG*[x]), that 
is to say, the number of solutions of G(x) = p, where each solution is counted 
with sign, which depends on the orientation. If the Jacobian condition is not 
satisfied, or if the map G is merely continuous, small perturbations of G are 
used, and one proves that the definition is the same no matter which particular 
perturbation was taken. In any case, the index is always an integer.) See, 
for instance, [47], Chapter 2, for details. The degree is a homotopy invariant, 
as long as the homotopy H satisfies that H(x,t) 7^ p for boundary x (above 
reference, property 2-7). Applied to the above data, this gives that 

deg(F,0) = deg(-a; > 0) = (-l) n ) 

the last equality by exercise 2-3 in that reference. Since F never vanishes on 
S p , there is some e > so that |^(a;)| > e f° r a h x € S p . It follows that 

F (x) ^ y for all y g B £ (0) and x € S p , 

so the degree deg(F, y) is well-defined for all such y. Since the degree depends 
continuously on y (property 2-8 in the above reference), and it is an integer, it 
follows that 

deg(F, 2/ ) = deg( J F,0)^0 

for all y € B e (0). But nonzero degree implies that F(x) = y has at least one 
solution (property 2-15 in the reference), as required. I 

We can now complete the proof of Theorem 22. Without loss of generality, 
we take a; = 0. Considering the obtained closed- loop system, we must show 
that, if a system 

x= f(x) 
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denned on some ball about the origin is asymptotically stable with respect to 
0, then the image of the mapping 

x i-> f(x) 

must contain a neighborhood of the origin. Pick a ball B p (0) which is included 
in the domain of attraction DClC E"., 

We apply the above fact, using the following homotopy, for x G $ p (0) and 
te[0,l]: 

f f(x) if * = 

-x if t = 1 



ff(t,x) := < 



|U U,i]-i| if < * < 1 



which connects / and —x. Note that Hit, x) ^ for all x 7^ and all t, since 
there cannot be nonzero equilibria or periodic points. 

At points of the form (t, x) with t > 0, H is the product of the continuous 
function 1/t with 

H (t,x) - x, 

where Ho is the homotopy used in the proof of Theorem 21, so it is continuous 
there. It only remains to show continuity at t = 0. That is, we need to establish 
that, for each x and each s > 0, there is some neighborhood W of x and some 
5 > such that 

\H(t,y) — f{x)\ < e whenever y G W and t < 6. 

Since for every s and y it holds that - (<j>(s, y) — y) = - L /((/»(r, y)) dr, 

1 _l 1 r s r s 

(<f>(s,y)-y)-f(x) = - {f(<P(T, y ))-f(x)}dT+ f(0(T,y))dT. 

8 s Jo Jo 

By continuity of f{4>{-, ■)), there are a neighborhood W of x and <5 > so that 

||/(0(r, y ))-/(x)||< £ /2 

whenever y € W and r < <5. Let M be a bound on the values of ||/(</>(r, y))\\ 
for y G W and r < 5. Thus, if s < (5 it holds that, with t = jjt, 

\H(y,t)-f(x)\<e/2 + M^- t (5.60) 

provided y G W and s = ^^ < 5. The proof of Theorem 22 is completed by 
picking a smaller 5 so that t < 5 implies that the second term in (5.60) is less 
than e/2. I 

Remark 5.9.18 The proof uses only the existence of k so that the closed-loop 
system is Lipschitz. Note also that, for any given neighborhood V of x°, the 
image /(V x IX) must contain a neighborhood of x°: this is clear from the proof, 
or applying the Theorem to the system restricted to the state-space V. □ 
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Exercise 5.9.19 Consider the following system, with X = IR 3 and IX = K 3 : 

xi = u 2 u 3 
xi = uiu 3 

X 3 = UlU 2 

and its equilibrium pair x° = 0, u a = 0. 

(1) Show that V(x) = \\x\\ is a control-Lyapunov function for this system, so 
the system is asymptotically controllable. 

(2) Prove that there is no possible local C 1 stabilizer for this system. □ 

Exercise 5.9.20 Consider a system without drift 

x = G(x)u 

that is, a control- affine system for which / = 0. (Recall, cf. Equation (4.22) 
and material after that, that controllability of such systems is equivalent to the 
accessibility rank condition.) We assume that m < n and G is smooth. Show 
that (with x° = 0, u° = 0): 

(1) If rankG(O) = m then there is no possible local C 1 stabilizer for this system. 

(2) Give an example where there is a global C 1 stabilizer. □ 

5.10 Notes and Comments 

Constant Linear Feedback 

See Chapter 2 of [212] for an extensive discussion of controller and controllability 
forms, as well as related topics such as their use in analog computer simulations. 

The Pole-Shifting Theorem has a long history. The proof for single-input 
systems, simple once the controller form is obtained, appears to have been 
discovered independently during the late 1950s. Kalman credits J. Bertram, 
ca. 1959, and Kailath credits also [329]. (In 1962, Harvey and Lee reported 
in [174] a weak version of the Theorem, asserting only stabilizability, as a con- 
sequence of the single-input case.) The general, multivariable, case, over the 
complexes, was first proved by Langenhop in [265] and independently by Popov 
ca. 1964, who discussed it in [321]. Over the real field, a complete proof was 
given by Wonham in [431]. The simple proof for general fields, by reduction 
to the single-input case, was given in [191]. The result that there is some Fi 
so that (A + BF\,b) is controllable is known as Heymann's Lemma; our proof 
is from [177]. Pole-placement problems for "singular" or "descriptor" systems 
Ex = Ax + Bu are discussed for instance in [94] . 

For a solution of Exercise 5.1.11, see [430]; for Exercise 5.1.12, see [212], 
Section 3.2. 

For any given pair (A, B) and desired characteristic polynomial x, there are 
(when m > 1) many feedback matrices F such that X A , RF = X- As a set of 
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equations on the nm entries of F, assignability introduces only n constraints, 
one for each of the coefficients of the desired characteristic polynomial. It is 
possible to exploit this nonuniqueness in various ways, for instance in order 
to impose other design requirements like the assignment of some eigenvectors, 
or the optimization of some criterion, such as the minimization of ||-F||. Fur- 
thermore, the Pole-Shifting Theorem does not say anything about the possible 
Jordan form of A + BF, that is, the possible similarity class of the closed loop 
A matrix. The more precise result is sometimes called Rosenbrock's Theorem, 
and is given in [331]; a purely state-space proof of this Theorem was given in 
[117] (see also [119]). 

Design issues in control theory depend on the "zeros" of a system as well as its 
poles (the eigenvalues of A). The book [212] discusses various notions of "zero" 
for linear systems, a topic that will not be discussed here. Recently there has 
been work characterizing system zeros using techniques of commutative algebra; 
see [436] and references therein. 

Feedback Equivalence 

The Brunovsky form originated with [77]; see also [224], on which our discussion 
is based (but note that some of the formulas in the original paper are incorrect). 

Feedback Linearization 

The original formulation of the feedback linearization problem was given in [71], 
for the restricted case treated in Exercise 5.3.18. The problem in the general 
form, including the multiple-input case (m > 1) is from [207] and [197], and is 
covered in detail in textbooks such as [199] and [311]. We restricted attention to 
the single-input case because this is sufficient for illustrating the techniques but 
is technically much simpler than the general case. (In general, instead of A n and 
b n in the controller form 5.3, one must base the proof on the Brunovsky form 
and Theorem 14. Frobenius' Theorem must also be generalized, to deal with 
nested families of distributions A 1 C A 2 C . . . rather than single distributions, 
as part of the proof of the multiinput case.) Exercise 5.3.19 can be solved 
directly, not using feedback linearization ideas; see for instance [311], Theorem 
5.3. Example 5.3.10 is from [378]. 

Disturbance Rejection and Relatively Invariant Subspaces 

The notion of (^4, B)-invariant subspace and the applications to solving systems 
problems, were proposed independently by [39] and [432]. It forms the basis 
of the approach described in [430], which should be consulted for a serious 
introduction to the topic of relatively invariant subspaces and many control 
applications. See [178] for a different manner of presenting these concepts and 
relations with other formalisms. 

There has been much research during the last few years attempting to gen- 
eralize these techniques to nonlinear systems. See, for instance, the textbooks 
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[199] and [311] for a treatment of that topic. The basic idea is to think of the 
space V not as a subset of the state space, but as what is sometimes called a 
"distribution" of subspaces in the tangent space to X. That is to say, one thinks 
of V as an assignment of a (possibly different) space V(x) at each x £ X. (See 
Definition 4.4.8.) Invariance then is defined with respect to the possible direc- 
tions of movement, and an analogue of Lemma 5.4.3 is obtained, under strong 
extra assumptions, from the classical Frobenius' Theorem on partial differential 
equations (cf. Theorem 10 (p. 175).) See also [166], [167], [194], [200], [257], 
[302], and [303] in this context, including discrete-time nonlinear versions. 

Stability and Other Asymptotic Notions 

There are a large number of references on stability, most of them in the classical 
dynamical systems literature. The paper [63] gives an excellent bibliographical 
survey of those results on stability that are of major importance for control. 
Related to stability questions in feedback systems, it is worth pointing out that 
recently there have been studies of control laws in relation to chaotic and other 
highly irregular behavior; see, for instance, [29] and [341], as well as the relevant 
papers in the book [273] and the very early work already discussed in [213]. 

Attractivity is, in general, not equivalent to asymptotic stability, contrary 
to what happens for linear systems (Proposition 5.5.5); for a counterexample, 
see, for instance, [169], pages 191-194. The weaker notion of attractivity is less 
interesting, partly because positive results usually will establish the stronger 
property anyway, and also because of considerations that arise in input/output 
design. It is also possible to introduce notions associated to simply "stability" 
(as opposed to asymptotic stability). For differential equations (no controls) this 
is defined as the property that states near x° should not give rise to trajectories 
that go far from x . Except in the context of proving an instability result, we 
did not discuss this notion. 

Stable and Unstable Modes 

For a solution of Exercise 5.6.6 see [430], Chapter 4. That book also has many 
applications of the concepts discussed in this section. 

Lyapunov and Control-Lyapunov Functions 

There are various results which provide converses to Theorem 17, at least for 
systems with no controls. The book [49] covers abstract dynamical systems. For 
continuous-time systems x = fix), with /(0) = and evolving in W 1 , and under 
weak assumptions on / (locally Lipschitz is sufficient), Massera's Theorem, given 
in [296], asserts that, if the system is asymptotically stable with respect to 
then there is a smooth Lyapunov function V , which satisfies the infinitesimal 
decrease condition V(x) = LfV(x) < for all x ^ 0; if the system is globally 
asymptotically stable, then a proper V exists. In other words, Lemma 5.7.4 is 
necessary as well as sufficient in the case of systems with no controls. For other 
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proofs see also [169], [262] (allows continuous /; even though solutions may 
not be unique, a notion of asymptotic stability can be defined in that case), 
and [428]. The reader may also find a proof in [279], which extends Massera's 
Theorem to smooth Lyapunov functions for systems subject to disturbances 
(robust stability), and is presented in the more general context of stability with 
respect to sets rather than merely equilibria. 

For control systems, the situation with converses of Lemma 5.7.4 is far more 
delicate. The existence of V with the infinitesimal decrease property is closely 
tied to the existence of continuous fedback stabilizers; see [23], [372]. If differ- 
entiability is relaxed, however, one may always find control-Lyapunov functions 
for asymptotically controllable systems: see the result in [363], and the recent 
applications to discontinuous feedback design in [92]. 

In any case, all converse Lyapunov results are purely existential, and are 
of no use in guiding the search for a Lyapunov function. The search for such 
functions is more of an art than a science, and good physical insight into a 
given system plus a good amount of trial and error is typically the only way 
to proceed. There are, however, many heuristics that help, as described in 
differential equations texts; see, for instance, [334]. Another possibility is to 
build control-Lyapunov functions recursively, via "backstepping" as discussed 
in Lemma 5.9.5 and pursued systematically in [259]. 

For Exercise 5.7.17, see e.g. [323]. 

Example 5.7.12 implies that there exists a bounded feedback of the type 
u = 9{Fx) which stabilizes a double integrator. The result might not be sur- 
prising, since we know (cf. Exercise 3.6.8) that the double integrator [A has 
all eigenvalues at zero) is null-controllable using bounded controls. It is a bit 
surprising, on the other hand, that, for the triple integrator x\ = X2, ±2 = £3, 
±3 = u, while still true that every state can be controlled to zero using bounded 
controls, it is in general impossible to find any globally stabilizing feedback of 
the form u = 6{Fx), with 9 a saturation function; see [150]. On the positive 
side, however, slightly more complicated feedback laws do exist in general, for 
stabilizing linear systems with saturating inputs; see [392]. 

The example in Exercise 5.7.21 is due to [295]. In the adaptive control 
literature one finds extensive discussions of results guaranteeing, for slow-enough 
time variations, that a time- varying system is stable if the "frozen" systems are. 



Linearization Principle for Stability 

Even though, as discussed in Section 5.9, the linearized condition in Theorem 
19 is sufficient but not necessary, this condition does become necessary if one 
imposes exponential stability rather than just asymptotic stability; see, for in- 
stance, [372] and references therein, as well as [157]. 

Theorem 20 is basically Lyapunov's Second Theorem on Instability, proved 
in 1892; it is often referred to (in generalized form) as "Chaetaev's Theorem." 
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Introduction to Nonlinear Stabilization 

The question of existence of smooth and more generally continuous feedback 
laws in the nonlinear case is the subject of much current research; it is closely 
related to the existence of Lyapunov functions and to the existence of smooth 
solutions to optimal control problems. See, for instance, the survey papers [372] 
and [101], as well as [1], [4], [23], [27], [54], [72], [92], [99], [100], [109], [186], [187], 
[210], [238], [268], [294], [339], [342], [355], [403], [407], and [443]. The recent 
textbook [259] covers many questions of nonlinear feedback design. Discontinu- 
ous feedback of various types is, of course, also of interest, and is widespread in 
practice, but less has been done in the direction of obtaining basic mathematical 
results; see, for instance, [385] for piecewise analytic feedback, [362] for piece- 
wise linear sampled feedback, [353] for pulse-width modulated control (a type of 
variable rate sampling) , and [92] for a notion of feedback based upon the idea 
of sampling at arbitrarily fast rates. A major recent result in [99] showed that 
time-varying smooth feedback stabilizers exist for a large class of systems (see 
also the related work, and alternative proof, in [374]). There is also an extensive 
engineering literature, e.g., [406], on sliding mode control, in which discontinuity 
surfaces play a central role. 

The line of work illustrated by Proposition 5.9.1 and Exercise 5.9.2 is based 
on [202], [210], and [268]. Exercise 5.9.4 is from [8]. 

The assumption in Lemma 5.9.5 about the existence of a suitable Lya- 
punov function is redundant, since stabilizability implies it (Massera's Theorem, 
quoted above). The proof of this Lemma was given independently in [80], [249], 
and [403]. An alternative proof of the same result about stability of cascade 
systems is given in [368], using only elementary definitions. A local version of 
the result has been known for a long time; see, for instance, [415]. For closely 
related work, see also [250]. 

Proposition 5.9.10 is essentially from [23]. The proof that we gave, based 
on the use of the "universal" formula (5.56), is from [369]. The result in [23] is 
far more general than stated here, however, especially in that it includes con- 
sideration of control- value sets IX which are proper subsets of W n . For universal 
formulas that apply to more general sets IX, and for applications, see e.g. [199], 
[259], [278]. The material that centers around Proposition 5.9.10 can be stated 
in necessary and sufficient form, because the existence of a stabilizing feed- 
back implies, via the converse Lyapunov theorem cited earlier, the existence of 
appropriate control-Lyapunov functions; see [23]. 

Theorem 22 was stated originally in [72]; see [443]. The proof given here is 
close to that for vector fields in [254]. See also [338]. 

There is also a wide literature regarding the stabilization of infinite dimen- 
sional linear systems; see, for instance, the survey article [324], as well as [113] 
and the general references cited for infinite dimensional linear systems theory. 



Chapter 6 

Outputs 



Except for the basic definitions given in Chapter 2, the other major ingredient 
in control theory, taking into account the constraints imposed by the impos- 
sibility of measuring all state variables, has not been examined yet. These 
constraints were illustrated in Chapter 1 through the example of proportional- 
only (as opposed to proportional-derivative) control. Section 1.6, in particular, 
should be reviewed at this point, for a motivational introduction to observability, 
observers, and dynamic feedback, topics which will be developed next. 

As discussed in Chapter 1 , a natural set of questions to consider when mea- 
surements are restricted revolves around the concept of observability. This deals, 
in some sense, with the inversion of the relation between states and outputs to 
recover states, a problem dual to that of controllability, which deals instead with 
the correspondence between controls and states. This duality will not be made 
explicit in general — there has been work framing it in the language of category 
theory — but the discussion in the first sections of this Chapter closely parallels 
the beginning of Chapter 3. Moreover, in the special but most important case of 
linear systems, the duality between control and observation takes a particularly 
elegant form, and it allows the application of results from previous chapters to 
problems of observability. 



6.1 Basic Observability Notions 

In all of the definitions that follow, E = (T, X, U, </>, y, h) is an arbitrary system 
with outputs. 

Recall from Definition 2.2.4 that for each initial state x° the i/o response of 
the initialized system (T,,x°) is the map A^ x o (denoted A^o, or even just A, if 
£ and/or x° are clear from the context), which assigns to each triple (r, a, lu) 
with u) admissible for x a the function value 



h(T,<f>{T,a,x°,uj)) 
261 
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representing the last output, at time r, due to applying the control u € U^' 1 "' 
when starting at the initial state x°. Especially when <j,t are fixed, we also 
denote A x o(t,<7, u>) as A CT o T (oj) or just A(u>). 

For instance, for the linear continuous-time time- varying system 

(6.1) 







x = A{t)x + B(t)u 






y = C(t)x 


one has that 






A^ T H = 


= V(r) = 


= C{t)$(t,(j)x + C(t) 



&(t,s)B(s)uj(s) ds , 

where $ denotes as usual the fundamental solution of x = A(t)x. 
Definition 6.1.1 Let x, z G X and o~,t G T , with a < r. 

• The control uj € 11^^' distinguishes between the events (x,a) and 
{z,o~) if it is admissible for both and 

A7M^r(w). 

• The events {x,cr) and (z,o~) are distinguishable on the interval [a, t], 

or in time (at most) T = r — a , if there is some t € [er, r] and some 
uj € Iv ' 1 ' that distinguishes them. If (x,o~) and (z,o~) are distinguishable 
on at least one interval, then they are just called distinguishable. 

• The states x and z are distinguishable by a given control, or in time T, or 
just distinguishable, if there is some a G T so that the respective 'property 
holds for the events (x,o~) and (z,a). □ 

Distinguishable events (x, a) and (z, a) are therefore those that can be sep- 
arated by applying some control, in the sense that if the system is initially in 
state x at time a it will behave differently than if it started in state z at time 
a. They are indistinguishable (i.e., not distinguishable) if every control leads 
to the same output. If one thinks of a system as a "black box" into which 
inputs can be applied and from which outputs emerge, distinguishability of in- 
ternal states — or events, if time is of importance — is precisely the possibility 
of differentiating among them on the basis of input/output experiments. 

Note that distinguishability in time T implies distinguishability in any larger 
time T" > T, by definition. Also, 

h(o~, x) t^ h(o~, z) 

is equivalent to distinguishability in time because 

Ar (o) = Ka,x) ± h(a,z) = \°f (o) , 

where o is the control of length zero. One may call this "instantaneous distin- 
guishability." 
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Example 6.1.2 Consider the states x = 1 and z = —1 for the continuous-time 
system 

2 2 

x = x , y = x 

having X = R and IX a one-point set. On any interval [0,t], < t < 1, the 
outputs are 



and 



\0,t 

A -i 



\0,t 

A i 



1 



1-M 



l-t 



which are distinct; thus, 1,-1 are distinguishable in time t. (We omit "cj" 
from the notation for A since there are no controls.) On any interval [0,i] with 
t > 1, on the other hand, A x ' is undefined, so 1,-1 cannot be distinguished 
by a control of length larger than 1, but they are still distinguishable in any 
time larger than 1 according to our definition. Note that each pair of states 
X, z so that x + z ^ is instantaneously distinguishable by the measurement 
y = x 2 , but how the dynamical behavior is essential in separating any other 
pair of states x and —x. □ 

We use the notation (x, a) ~ (z, a) to indicate that the events (x,tr) and 
(z,o~) are indistinguishable on the interval [a, r] and just (x,er) ~ (z,o~) if they 
are indistinguishable. For states, we write x ~ z if x cannot be distinguished 
from z in time T and just x ~ z to indicate that they are indistinguishable. 
Observe that 

(x, a) ~ (z, a) iff (x, a) ~ (2, er) for all r > cr 

as well as 



and 



x ~ 2; iff (x, it) ~ (z, cr) for all tr, r so that t — a = T 



iff x ~ 2; for all T G 7^_ . 



Lemma/Exercise 6.1.3 Assume that E is a complete system. Show that in 
this case, on events and states, ~ and ~ are equivalence relations. □ 

Definition 6.1.4 The system E is observable on the interval \<J,t] if for 
every pair of distinct states x and z the events (x,a) and (z,a) are distinguish- 
able on the interval [a, r] . It is observable in time T if any two distinct states 
are distinguishable in time T and just observable if any two distinct states are 
distinguishable. □ 

As in Chapter 3, these notions simplify considerably for linear systems. Re- 
call that a linear system is one that is either as in Definition 2.4.1 or as in 
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Definition 2.7.2. By linearity of 4>{t, <t, •, •) (see Lemmas 2.4.2 and 2.7 '.4) and of 
h(t, •), it follows that 

K ,T {u) = h(r,(f)(T,a,x,uj)) 

is linear on (x,u>). From (x,u>) = (x, 0) + (0,w), it follows that 



A£>» = A^(0) + A£'» (6.2) 



for all pairs (x, uj), where denotes the control la = of length t — a . Equation 
(6.2) is sometimes called the superposition principle, because it expresses the 
fact that for linear systems the output from any initial state is the superposition 
(sum) of the zero-state response and the response to the zero input. 
In particular, for any two states x, z, if 

A£» + K' T W) 

for some oj, then also 

AT (o) + K lT (o) 

and so 

for all to. In particular, the events (x, a) and (z, a) are distinguishable on [a, t] if 
and only if distinguishes them on some subinterval [a, t] . Similar remarks apply 
to the distinguishability of states. The equality A^ 2 (0) = A^ ,T (lu) — A^' T (w) 
implies: 

Lemma 6.1.5 For linear systems, (x,a) and (z,a) are indistinguishable on 
[a, t] if and only if 

K'lM = o 

for all t G [a, r]; that is, (x — z,a) is indistinguishable from (0, it) using zero 
controls on [it, t). Similarly, any two states x, z are indistinguishable if and only 
if x — z is indistinguishable from 0. □ 

Corollary 6.1.6 For linear systems, observability on the interval [a, r] is equiv- 
alent to 

T 

P|ker (x^ A£'*(0)) = 0. □ 

For linear systems observable on [a, r] a single experiment, namely applying 
the zero control (on increasing intervals), serves to separate all possible pairs. 
This is in general false for nonlinear systems, as illustrated by the following 
example using a finite discrete-time system. 

Example 6.1.7 Consider the discrete-time time-invariant finite system with 

X = {a, b, c, d} , U={u,v}, y = {a,/3}, 



6.1. Basic Observability Notions 



265 



local- in-time transition function V given by the following table: 



V 


u 


V 


a 


c 


c 


b 


c 


b 


c 


a 


a 


d 


d 


a 



and output function h(a) = h(b) = a and h{c) = h{d) = (3. The pairs 

{a,c},{a,d},{b,c},{b,d} 
are instantaneously distinguishable. To distinguish 

{a,b} 
one may use any sequence starting with v and to distinguish 

{c,d} 

any sequence starting with u. In summary, every pair of distinct states can be 
distinguished in time 1, and the system is observable in time 1. 

There is however no single "universal" sequence ui € 11^' ' T > so that for every 
pair x t^ z it would hold that 

A*><M M) ) ^ A^M M) ) 

for some t (that is allowed to depend on x and z). This is because if uj starts with 
u, then all outputs when starting at a or b coincide, since the state trajectories 
coincide after the first instant, but if u> starts with v, then lu cannot separate c 
from d. □ 

This Example illustrates the "hypothesis testing" character of observability: 
For each pair of states there is some experiment that allows one to distinguish 
between them, but there is in general no "universal" experiment that allows a 
diagnosis of the initial state. The notion introduced next is somewhat better 
behaved in that sense. 



Final-State Observability 

Note that in Example 6.1.7, even though the initial-state information may have 
been lost, it is still the case that after applying a suitable input sequence and 
measuring the outputs, the final state may be determined completely. If the 
ultimate objective is to control the system to a given state, this knowledge is 
sufficient. As an illustration, assume that it is desired to bring the system to 
x = a. This is a reasonable objective, since every state can be controlled to a, 
but the goal now is to choose the controls based only on output measurements. 
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Consider the control on [0,2) with w(0) = u and w(l) = v. The outputs at 
times 0, 1, 2 constitute one of the sequences 

a(3a f3(3a (5a(3 , 

and if either of the first two happened — corresponding to the possible initial 
states a, b, or d — then we know that the final state, the state at time 2, is 
already a, which is as desired. If the last happens — initial state was c — the 
final state is known to be c. In either case, all future behavior of the system is 
completely known. Control to a is achieved in the last case simply by applying 
one further input, either u or v. 

This discussion motivates the next definition, totally parallel to that of dis- 
tinguishability. The terminology that we use is not standard. Since as proved 
later this concept does not give anything essentially new for continuous-time 
systems, this subsection can be skipped by the reader interested only in contin- 
uous-time. 

As in Chapter 2, Remark 2.2.2, A denotes the entire output corresponding 
to a control u>, that is, 

A^M(i):=A^H kit) ) (6.3) 

for each t G [a, t] and each state x, and we drop x or a, r when they are clear 
from the context. 

Definition 6.1.8 The control u) € U' oyr ) final-state distinguishes between 
the events (x,a) and (z,o~) if uj is admissible for both x and z and either 

. X^(cj) ± \YW), or 

• </>(t, a, x,u>) = </>(t, <t, z, u>). □ 

As with distinguishability, if there exists one such u) on some interval [a, t) so 
that t < r, then {x, a) and (z, a) are final-state distinguishable on the interval 
[a, t] or in time r — a. If (x, cr) and (z, a) are final-state distinguishable on 
some interval [a, r], then they are just called final-state distinguishable. For 
states x and z, they are final-state distinguishable by a given control, or in 
time T, or just final-state distinguishable if there is some a G T for which the 
respective property holds for the events (x, a) and (z, a). A system £ is final- 
state observable on an interval [a, r] if for every pair of states x and z the events 
(x,a) and {z,o~) are final-state distinguishable on the interval [<j,t\. It is final- 
state observable in time T if any two states are final-state distinguishable in time 
T, and just final-state observable if any two states are final-state distinguishable. 

It is immediate from the definition that distinguishable states are also final- 
state distinguishable. Thus, observable systems are always final-state observ- 
able. The converse is not true, as illustrated by the discrete-time system 

x+ = 0, y = 0, 
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with X = U = y = R. No two states are distinguishable, and hence the system is 
in particular unobservable, but every control (of length at least one) final-state 
distinguishes, since the state is zero after one instant, so the system is final-state 
observable. However, for continuous-time systems the concepts are equivalent: 

Proposition 6.1.9 A continuous-time system is observable on [a, r] (or ob- 
servable in time T, or observable) if and only if it is final-state observable on 
[a, t] (respectively, in time T, final-state observable). 

Proof. Necessity always holds, so pick a final-state observable system and any 
two distinct events (x,a) ^ (z,a). Let lo be so that it final-state distinguishes 
the events (x,cr) and (z,a); without loss of generality, assume lo € W a,T > . If 
X^ t (lo) / \°' t (lo), then some restriction W[ a ,t) distinguishes between them, and 
there is nothing left to prove. So assume that 

(j)(T,a,x,u)) = <f>(T,a,z,u>). 

Let t;(t) = 4>(t,a,x,uj) and £(£) = <p(t, a, z,u>). Then £ and C are two solutions 
of x = f(x, lo) defined on all of [it, t] and having the same value at time t = t. 
By uniqueness of solutions, £ = £, contradicting the assumption that £(c) = 
x t^ z = C( cr )- This gives the implication on any fixed interval [a, r], and the 
other implications follow from here. I 

As discussed earlier, observability requires multiple experiments, except for 
linear systems and some other restricted classes of systems. But if a system is 
final-state observable, it is often possible to prove that it is final-state distin- 
guishable using single (long-enough) experiments, as in the following exercise. 

Exercise 6.1.10 Assume that £ is a final-state observable time-invariant dis- 
crete-time complete system for which card X < oo. Show that there exists some 
T > and some fixed control uj of length T so that lo final-state distinguishes 
every pair of states. {Hint: Consider any control for which the set of pairs of 
states (x, z) that are final-state indistinguishable by uj is of minimal possible 
cardinality. ) □ 

Continuous-Time Systems 

There is an analogue of the discussion in Section 3.9, dealing with the sufficiency 
of various classes of controls in testing observability for continuous-time systems. 
For any fixed <r, r, let A be any subset of £^ (a, r) so that for each uj € £?? 
there exists a sequence of elements 

uj J ' € A , lo^ j ' — > lo pointwise a.e. , 

which is equibounded (all values in a fixed compact). 

Examples of sets A that satisfy the above conditions are discussed in Remark 
C.1.2 in Appendix C. They include analytic controls when IX is an open convex 
subset of R m . 

The following observation then is very useful. 
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Proposition 6.1.11 If (xi,a) and (#2, &) are distinguishable, then they can 
be distinguished by a control in A. 

Proof. By Theorem 1 (p. 57), Part 2(i), and the continuity of h, it follows 
that, if a; € XV- (7 ' T > is admissible for some x, and if u>v) — > & pointwise almost 
everywhere and the sequence is equibounded, then u>v) is also admissible for x 
for large j, and 

Thus, if u> distinguishes two events and u>v) — > w for some equibounded sequence 
in A, then for large enough j one of the controls in the approximating sequence 
already distinguishes. I 

An interesting consequence of Proposition 6.1.11 is the fact that, for analytic 
systems, observability can be achieved almost instantaneously. The analogous 
controllability result is of course false, except for linear systems (cf. Remark 
3.7.9). 

Proposition 6.1.12 Let E be a continuous-time system over EC = K, 

x = f(t,x,u), y = h(t,x), 

for which the maps / and h are real-analytic in all their arguments, 11 and y 
being assumed to be open subsets of Euclidean spaces K m and W, respectively, 
with IX convex. Assume that (x,a) and (z,a) are distinguishable on [a, r] and 
pick any 5 > 0. Then, these events are distinguishable in time 5. 

In particular, if E is observable on [a, r], then it is observable in arbitrarily 
small time S > 0. 

Proof. The two states x and z can be distinguished by an analytic control lu 
on [<7,t], by Proposition 6.1.11. Let £ = ip(r, a, x,uS) and C, = ^(t,<t,z,w) be 
the ensuing paths and let 

a(t) := K> T (uj)(t) = h(t, £(*)) , 0(t) := A? T (w)(t) = /»(*, CW) • 

These are analytic functions of time, because /i as well as each of £ and C are (cf. 
Proposition C.3.12 in Appendix C). Thus, if these functions were to coincide 
on [0,8], then they would be identically the same, contradicting the fact that 
they differ at t = r. I 

Exercise 6.1.13 Prove that if E is a continuous-time system as in Proposition 
6.1.12, then indistinguishability is an equivalence relation. □ 

6.2 Time-Invariant Systems 

Throughout this section, unless otherwise stated, E is an arbitrary time-invar- 
iant system with outputs. 
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For each fixed t £ T + we denote 

/* := {(x, z) Glxl|i~z}, 
that is, the relation ~ seen as a set of ordered pairs. Note that 



J* C I s 



for each pair s < t. We also let 



rv< 



teT+ 

denote the set of indistinguishable pairs. The following can be seen as a dual 
statement to the one in Lemma 3.2.6: 

Lemma 6.2.1 If there are any S, T £ T + , T > 0, such that 

I s = I s+T , 
then necessarily I s = I. 
Proof. It will be enough to establish that 

I s = I s+kT (6.4) 

for all positive integers k, since then for each t £ T + there is some k so that 
S + kT > t and therefore I s = I s + kT C P . We prove (6.4) by induction on k, 
the case k = 1 holding by hypothesis. Assume then that (6.4) holds for k — 1 
and pick any 

(x,z) £l s = I s+T . (6.5) 

We want to prove that (x, z) £ I s + kT . If this were not to be the case, then there 
would be some to £ W ^' that distinguishes between them, with t < S + kT. 
It cannot be the case that t < T, since that would contradict the fact that 
(x, z) £ I + . So we may consider the control 

^i : = H[0,T) i 

which must be admissible for both x and z because uj was. Let 

x\ := (t>(T,0,x,u>i), z\ := (j>(T,Q,z,u\) . 

The control 

w 2 := w\[ T> t) 

distinguishes x\ and Z\, in time t — T < S+(k— 1)T, so by inductive hypothesis 
these states can also be distinguished by some control lu^ of length at most S. 
Finally, introduce the concatenation 
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which is a control of length at most S + T distinguishing x and z. The existence 
of such a control contradicts (6.5). I 

The following two results are analogues of Lemma 3.2.4 and Corollary 3.2.7, 
respectively. 

Corollary 6.2.2 Assume that E is a complete discrete-time system and 1 < 
card (X) = n < 00. Then 

j = jn-2 f 

and in particular E is observable if and only if it is observable in time n — 2. 
Proof. Consider the sequence of sets 

jn-l g jn-2 C ... C 7° C Xx X. 

The inclusions cannot be all strict. Indeed, recall (Lemma 6.1.3) that, since E 
is complete, indistinguishability in time i is an equivalence relation. Let U be 
the number of equivalence classes under ~ ; strict inclusions would imply 

ln-1 > ln-2 > ■ ■ ■ > h > 1 ) 

which is impossible since all U < n. Thus, either 1° = X x X, i.e., h is constant 
and therefore / = X x X, or we are in the situation of Lemma 6.2.1. I 

Exercise 6.2.3 Show that, for arbitrary finite discrete-time systems, not nec- 
essarily complete, 

Give an example to illustrate that the formula I = J" -2 may not hold if the 
system is not complete. □ 

If E is a linear system and T G T + , we let 

T 



T :=O|A^(O)=O} 



so that ir~ z if and only if x — z € Ot- In this case, a strict inclusion 

7 T C/ S 

is equivalent to 

dim Ot < dim Os , 

so there cannot be any proper chain 

T t n c I t n . 1 c _ _ c jto c Xx X, 

and this implies, by the same argument as that used to prove Corollary 3.2.7: 



6.2. Time-Invariant Systems 271 

Corollary 6.2.4 Assume that E is a linear system of dimension n < oo. Then: 

(a) If E is discrete-time, / = I n . 

(b) If E is continuous-time, 1 = 1 for all (5 > 0. 

In particular, observable finite dimensional discrete-time systems are observable 
in n — 1 steps, and observable finite dimensional continuous-time systems are 
observable in arbitrarily small time. □ 

The continuous-time part is of course also a consequence of Proposition 
6.1.12. Observe that for a discrete-time linear system over a finite field of q 
elements, Corollary 6.2.2 gives an estimate of q n — 2 steps, while using linearity 
gives the far smaller estimate n — 1 . 

For discrete-time linear systems, C„_i = kerO„(^4, C), where for each k we 
are denoting 

( ° \ 



O k (A,C) 



CA 



\CA fe -V 



(6.6) 



and this kernel is zero precisely when the transposed matrix has full row rank, 
from which the following fact follows (denoting O n simply as O): 

Theorem 23 The n- dimensional discrete-time linear system E is observable if 
and only if 

rankO(A, C) = n, 

or equivalently, if and only if (A' , C) is controllable. □ 

For continuous-time linear systems, 

O e = {x\ Ce tA x = for all t € [0, e]} , 

which is equivalent, by analyticity of Ce* x, to the set of conditions 

CA l x = 0, i = 0,1,2,... , 

or equivalently, because of the Cayley-Hamilton Theorem, to these same condi- 
tions for i = 0, . . . ,n — 1. Thus, again observability is equivalent to the kernel 
of the matrix in (6.6) being trivial for k = n, and we conclude as follows. 

Theorem 24 The n-dimensional continuous-time linear system E is observable 
if and only if 

rankO(A, C) = n, 

or equivalently, if and only if (A', C) is controllable. □ 
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By the two Theorems, there is no ambiguity in calling a pair of matrices 
{A, C) with A e K nxn and C e K.P xn an observable pair if 0(A, C) has rank 
n. In general, all the material in Section 3.3 dualizes to results about observable 
pairs. For instance, the Hautus controllability condition becomes: 



(A, C) is observable iff rank 



XI- A 
C 



n for all A € K (6.7) 



and the set of all observable pairs is generic, in the sense of Proposition 3.3.12, 
among all pairs. 

The Kalman controllability decomposition in Lemma 3.3.3 dualizes to an 
observability decomposition: If rankO(A, C) = r < n, then there exists a T G 
GL{n) such that the matrices A := T~ l AT and C := CT have the block 



structure 



Ai 

A 2 A 3 



C=(Ci 0) 



(6.8) 



where A\ is r x r, C\ is p x r, and (Lemma 3.3.4) the pair (Ai, C\) is observable. 
As before, if r = 0, the decomposition is trivial, and A\, 0, A 2 , C\ are not there. 

As in the controllability case, two pairs (A, C) and (A, C) are said to be 
similar if such a T exists. 

This decomposition corresponds, using continuous-time to illustrate, to a 
change of variables 

x(t) := Tz{t) 

under which the equations x = Ax, y = Cx are transformed into 

z\ = Aizi 
z 2 = A 2 zi + A 3 z 2 
V = Cizi , 

where Z\, z 2 are r- and (n— r)-dimensional respectively. Thus, the z 2 coordinate 
does not affect the output, even indirectly. 

Exercise 6.2.5 Refer to Exercise 3.2.12. Show that using the altitude h as the 
measurement map, the system E is observable. Show that using instead any 
other coordinate as an output (for instance, </>, which can be measured with a 
gyroscope) would not result in observability. □ 

Exercise 6.2.6 Refer to Exercise 3.2.13. 

(a) Show that the system is not observable if one uses the pendulum angle 
£3 = <f> as the measurement map. Provide an example of two indistin- 
guishable states, and explain intuitively why they are indistinguishable. 

(b) Show that using the cart position x\ = 5 as the measurement map, the 
system E is observable, as long as m > 0. However, when m = 0, which 
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models the case where the pendulum mass is negligible compared to that 
of the cart, the system is not observable. (Why is this difference more or 
less clear intuitively?) 

(c) Prove that if both 5 and 4> are measured, i.e. 

/l 
'"^0010 

then the system is observable even if m = 0. □ 

Observability Under Additional Structure 

Many possible nonlinear generalizations of Corollaries 6.2.2 and 6.2.4 are avail- 
able when additional algebraic assumptions are made on the system structure. 
The following discussion is meant to illustrate this fact through some examples. 
Consider complete discrete-time (still time-invariant) polynomial systems. 
These are systems with X = E™, IX = R m , V = W, and equations 

x = V(x, u) , y = h{x) , 

for which the maps V : K" +m — > K" and h : K" — * K p are polynomial. 

Proposition 6.2.7 Let E be as above. There exists then a finite set of controls 
Wi, . . . , u>k such that, for any pair of states x and z, these states are distinguish- 
able if and only if one of the controls u>i distinguishes them. In particular, if E 
is observable, then there is some finite time T so that E is observable in time T. 

Proof. For each possible control la let A w be the polynomial in 2n variables 
corresponding to the function 

A u (x, z) := X x {uj) - X z (qj) . 

In particular, A (cc, z) = h{x) — h{z). Observe that lu does not distinguish 
between the two states x and z precisely when A. u (x, z) = 0. 

From the Hilbert Basis Theorem, we know that there exists a finite set of 
controls wi,. . . ,u>k so that for each other u there are polynomials p\ , . . . , pk 

with 

fc 

A u (x,z) = y^ j p i (x,z)A uli (x,z) . 

This implies that, if for any given pair (a; , z ) of states it holds that 

A UJi {x°,z o ) = for alli= l,...,k, 

then, for the same pair, necessarily A UJ (x , z ) = for every other control, and 
the conclusion is established. (Recall that the Hilbert Basis Theorem asserts 
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that every ideal in the polynomial ring K[xi, . . . , x n , z\, ... , z n ] is finitely gen- 
erated. We apply this to the ideal generated by the A u 's. Each element of this 
ideal is a finite linear combination of the elements A w , so each member of a 
finite generating set can in turn be written in terms of a finite number of the 
A„'s.) I 

Exercise 6.2.8 Show that the result does not extend to analytic systems by 
considering the system 

+ 1 

X = —X , V = SIM , 

2 
which is observable but not in finite time. □ 

Exercise 6.2.9 There is a continuous-time version of the above result. Con- 
sider continuous-time polynomial systems 

x = f(x,u), y = h{x), 

where X,U,y,/i are as above and / is polynomial. We assume completeness. 
Show that, again, there exists then a finite set of controls uii, . . . ,u>k such that, 
for any pair of states x and z, these states are distinguishable if and only if one 
of the controls tOi distinguishes them. [Hint: Use analytic controls and argue in 
terms of the corresponding derivatives y(0),y(0), . . . that there must exist for 
each such control a polynomial A w so that x and z are indistinguishable by to if 
and only if A w (x, z) = 0. The Hilbert Basis Theorem is needed for that. Now 
apply the Basis Theorem again.) □ 

Abstract Duality 

The duality between controllability and observability can be formalized in var- 
ious manners for nonlinear systems. More precisely, a form of duality holds 
between reachability from a fixed initial state and observability. For linear 
systems, reachability from the origin is equivalent to complete controllability, 
and this explains the use of "controllability" in the linear case. The following 
exercise illustrates one abstract — and trivial — version of this duality. 

Exercise 6.2.10 Let E be a complete time-invariant system with outputs and 
cardy > 2. Consider the system 

£ := (T,X,1X,0) 

defined as follows. Its state space X is the set of all maps X — > y, and for each 
uj G lt' CT ' r ) and each state a : X — > ^ of X, 

4>{t, <r,a,uj) 

is the map 

x \— > a((f>(T, a, x,uj)) , 
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where uj is the time-reversed control u>(t) := Lu(a + r — t). Note that the output 
map h can be seen as an element of X. Prove that if £ is reachable from h, i.e., 

TZ(h) = X, 

then £ is observable. □ 

Sampled Observability 

Let £ = (A, B, C) be a linear time-invariant continuous-time system with out- 
puts over IK = K. Consider the situation where measurements of the state of the 
system are only possible at discrete instants t = kS, for integers k, where S > 
is a fixed number, called the sampling interval. This situation was discussed 
in Section 1.3 on digital control, and a discrete-time system with outputs, £ui, 
which models the system obtained by disregarding the intermediate observa- 
tions, was introduced in Definition 2.10.1. In analogy with the discussion in 
Section 3.4 we ask when is £ 5- sampled observable, that is, when is £ui observ- 
able as a discrete-time time-invariant linear system? Note that 

X [s] = (e SA ,A^B,C), 

so observability is equivalent to controllability of 

(e SA \C) 

and therefore (5-sampled observability holds provided that (A' , C") is <5-sampled- 
controllable (Lemma 3.4.1). In particular, if 5 satisfies the spectrum conditions 
in Theorem 4 (p. 102), then observability of (A,C) implies (5-sampled observ- 
ability: 

Proposition 6.2.11 Assume that £ is observable and that 

5{X - fi) =£ 2/bri, fc = ±l,±2,..., (6.9) 

for every two eigenvalues A, /i of A. Then £ is also <5-sampled observable. □ 

Thus, again using the terminology in Section 3.4: 

Observability is preserved if the sampling frequency is larger than 
twice the largest frequency of A 

(cf. Lemma 3.4.4). 

Exercise 6.2.12 Let Ui,i = 1, . . . , k be k different positive real numbers. Show 
that there is some continuous-time time-invariant linear system with outputs 
and no inputs £ = (A,0,C) such that: 

• £ is observable, and 
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• for each set of 2k real numbers a», tpi, i = 1, . . . , k, there is some initial 
state x so that 



A x ' = 77(f) = 2> a i sin(2TTuJit + ifi) 



for all t > 0. 

Conclude from the above discussion that, if 

1 

— > 2 max \u!i\ , 

i—l.... : m 

then the complete function 77(f) can be recovered from the values 

r,(0),r,(6),r,(26),... 
for every set of o»'s and ip^s. □ 

6.3 Continuous-Time Linear Systems 

Pick any finite dimensional continuous-time linear system with outputs but no 
inputs E = (A(t),0,C(i)). Its adjoint S* is the system 

x(t) = -A(t)*x{t) + C(t)*u{t) . (6.10) 

Corollary 3.5.9 asserts that controllability of S* on an interval [a, r] is equivalent 
to there not being any nonzero solution £ of the equation x = — (— A*)*x = Ax 
on [cr, r] for which C(t)£(t) = 0. But then Corollary 6.1.6 gives: 



S is observable on [<j, r] <^=> S* is controllable on [a, r] 



Another way to state this adjointness is in terms of the operator 

L: n = Cl(a,r) -> X = K™ : wh / k(t)*U)(t) dt (6.11) 

«/ (J 

which was considered in Section 3.5, where 

fc(«) = C(a)*(*,r) ) (6.12) 

that is, the operator corresponding to the input-to-state map for the adjoint 
system. It was proved in Section 3.5 that L is onto (E* is controllable on the 
interval [a, r]) if and only if L* is one-to-one, where 

(L*x)(t) = C(t)$(t,T)x = C(t)$(t,a)$(a,T)x. 
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But this mapping is one-to-one precisely when there is no nonzero x so that the 
trajectory with initial state $(cr, t)x gives a zero output, that is, if the original 
system is observable. 

This adjointness permits applying the results of Section 3.5 to observability 
problems for continuous-time linear systems. For example, observability on 
[a, t] is equivalent to the positive definiteness of 



W = $(s,r)*C(s)*C(s)$(s,r)ds, (6.13) 

which in principle can be checked numerically. 

Least-Squares Observation 

The optimal-norm property of the pseudoinverse of the input-to-state operator 
in Section 3.5 dualizes into an optimal-observation property. This is discussed 
next, after some general remarks about pseudoinverses. 

In general, consider a bounded linear operator L from a Hilbert space tt 
into a finite dimensional space X, as in Section 3.5. Assume from now on that 
L is onto, so that the conclusions of Corollary 3.5.3 hold: The adjoint L* is 
one-to-one, and 

W : X^ X, W := LL* 

is positive definite. The pseudoinverse 

L* := L*W~ X : X^fl 

was defined in Equation (3.20), and it was shown that it provides the minimum- 
norm solution of Llu = x. The next result proves that its adjoint 

(L*y = w- 1 L-.n^i 

gives the least-squares solution of L* z = lu. (For operators between finite dimen- 
sional spaces, both of these are particular cases of the pseudoinverse construction 
in Appendix A. 2 on singular values.) 

Lemma 6.3.1 For each lu g fi, let z := W~ 1 Luj. Then 

||L**- W ||<||L*<-w|| 

for all C € X, with equality only if C = z - 

Proof. Pick any u> € O. Observe that for all a; € X it holds that 

(L*x, L*Luj - w) = (6.14) 

because, as LL#L = L, this expression equals (x,Lu> — Luj). In particular, 
given any (6lwe may apply (6.14) to the element x = C — Z to obtain 

(L*( - L*Luj, L*Llu -lu) = 
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and therefore that 

\\L\-uj\\ 2 = \\L* C - L* Luj\\ 2 + \\L* Luj - uj\\ 2 
= \\L* C - L* Llu\\ 2 + \\L* z - lu\\ 2 , 

which establishes the desired inequality. Equality can hold only if 

L*( = L*Lu> = L*z, 

which, since L* is one-to-one, is equivalent to £ = z. I 

Now consider the system £ = (A(i), 0, C(t)) and an interval [a, r]. We 
introduce the reachability operator associated to the adjoint system £*, that is, 
L as in (6.11), so that k is as in (6.12), W is as in (6.13), 



Lr] = ®(s,T)*C*{s)r]{s)ds, 
and 

(L*C)(t) = C(tm,T)( = C(i)*(i,<7)S(<7,T)C. 

We now let the operator 

M : £ 2 p (a, T ) -> X 

be defined by the formula 



M r) := $(ct, t)W~ x f $(s, t)*C{s)*t]{s) ds 



which is obtained as the composition <&(cr, t)W^ 1 L. 

Using the "A" notation for the entire output function as in Equation (6.3), 
Lemma 6.3.1 gives that x = Mqtj is the least-squares solution of 

A x (0) = r] 

that is, for any given output function 77, M^rj is the state that minimizes the 
difference between this function and the one that would correspond to initial 
state x and control = 0. 

In general, for the system (6.1), let 

M(u;,7y) := M (77 - A (u;)) 

be the operator on J0 2 n (a, r) x Cp(&, t) that subtracts the effect of any given 
control lu on the output and then provides the best least-squares solution for 
this new control. 

The main result on optimal observation follows immediately from the pro- 
ceeding discussion: 
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Theorem 25 Assume that E = (A, B, C) is observable. Take any ui G >C^(cr, r) 
and any r\ G £p(c, r), and Zet x := M(u>, 77). Then, 

||Ax(w) -7?|| < ||A x (w) -7y|| 

/or aZZ i£l, wii/i equality only if x = x. □ 

This can be interpreted as follows: If the actual output is 

y(t) = C(t)x(t) + d(t) , 

where d(-) represents an unknown disturbance, then the estimate x provides the 
state that would have produced this output under the assumption that d was 
as small as possible. 

Exercise 6.3.2 Let E be a continuous-time linear system with outputs, and 
pick a < t G K. Consider the observability Gramian 



W (a,T) :-- 



<5>{s,o-)*C(s)*C(s)$(s,a)ds. 



Show: E is observable in [ct,t] if and only if W (o~, r) has rank n, and, in 
that case, the operator that computes the initial state x from the observation 
77(f) = C(t)<&(t, cr)x is given by: 



x = M077 = W {<t,t)- 1 
(This is a dual to Exercise 3.5.5.) 



<J>(s,ct)*C(s)*77(s) ds . 



□ 



Analytically- Varying Systems 

If (A,B,C) is so that A and C have real-analytic entries on some interval 
X = [a, t] , and if 1 G I is an arbitrary point in that interval, then observability 
on [a, t] is equivalent to 



(6.15) 



rank 


-C (*o)l 
Ci(t ) 

-Cfe(to)- 




= n for some A; 


where, inductively on i, 

C (t) :-- 


= C(i) 


and 









Q +1 (i):=-Q(i)A(i)--Q(i) 

at 

for i G X. This is simply what results from applying the criterion in Corollary 

3.5.18 to the adjoint system. The negative sign in the first term on the right 
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hand side comes from the adjoint equation. Equivalently, multiplying Ci by 
(— l) 4 , one could take 

Ci+i(t) = d(t)A(t) + j t d{t) 

for i > 1. 

Remark 6.3.3 Of course, the above rank condition can also be obtained di- 
rectly, simply by observing that, if y = Cx and x = Ax, then in the analytic 
case y = if and only if all its derivatives vanish at to , but these derivatives are 

y {i \to) = C i (t )x(t ), 

and unobservability is equivalent to the existence of some x(to) ^ such that 
this happens. □ 

Remark 6.3.4 More generally, if E is a linear system with outputs and the 
entries of B are continuous functions of time, one may define the adjoint as a 
system with outputs 

(-A(ty,c(ty,B(ty) 

with the same state space, control- value set y, and output-value set IX. Since 
(E*)* = E, one can also state the adjointness result as: E is controllable if and 
only if E* is observable. □ 



6.4 Linearization Principle for Observability 

This section provides a result showing that, if a linearization is observable, then 
the original system is locally observable in a suitable sense. 

Definition 6.4.1 The topological system E is locally observable about the 

state x° G X if there exists some neighborhood V of x° such that every state in 
V different from x° is distinguishable from x Q . □ 

Theorem 26 Assume that E is a continuous-time system over M. of class C , 
and let T = (£,w) be a trajectory for E on an interval X = [o~,t\. Then a 
sufficient condition for E to be locally observable about x° = £(c) is that E*[r] 
be observable on [a, t\. 

Proof. We first remark a general fact about continuous-time linear systems. 
We claim that, if E is a linear system which is observable on [a, t], then there 
exist t\, ■ ■ ■ , t n G [a, r] and integers i\, . . . ,i n such that the mapping 

x i-» (C il (h)^(ti,a)x, . . . ,C in (t n )^(t n ,a)x) 
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is one-to-one, where Cj is the jth column of C '. This can be shown as follows. 
Assume inductively that we have constructed subspaces X = Kq, . . . , K r , times 
t\, . . . ,t r € [er, t], and integers i\, . . . , i r , such that for r > 

K r = {x\C il (t 1 )®(h,a)x= ... = C ir (t r )®(tr,a)x = 0} 

and so that Ki has dimension n — i for each i. If r = n, then we have already 
obtained the desired i^'s. Otherwise, pick any nonzero x € K r . By observability, 
there is some t € [a, t] so that C(t)<fr(t,a)x ^ and hence some column, say 
the jth, of this which is nonzero. Letting t r +\ := t and i r +i := j provides the 
induction step. For r = n the claim follows. Apply this remark to the system 
E*[r] to obtain ti, ■ ■ . , t n and i\, . . . , i n . 
Consider now the mapping 

(3, x^E": ar^(^;* i 1 ( W i),...,A^K)) , 

where u)j is the restriction of lu to the interval [a, tj) and the subscript ij indicates 
the ijth coordinate of Aa;' 3 . The differential of this mapping, evaluated at x°, 
can be obtained by looking at each component. Consider any mapping of the 
type 

X — > R : x h-» X°'i(w) , 

for any control lu. Its differential at x a is the composition of the partial differ- 
ential of the mapping a in Theorem 1 (p. 57) with respect to x, evaluated at 
(x°,uj), with the differential of the fth coordinate of h, that is, the mapping 

x i-» Ci(t)®(t,<r)x 

for the linearization. 

We conclude that the differential of (3 is one-to-one, by construction of the 
ij's and the ij's. Thus, from the Inverse Function Theorem, the map (3 is 
one-to-one in a neighborhood of a; , as wanted. I 

Of course this condition is far from necessary, as the example 



x = 0, y = x 3 



(and x° = 0) shows. 



Remark 6.4.2 There is a higher order test for observability as well. Here we 
only present a brief outline of this test. Assume given a continuous-time system 
affine in controls, 

m 

x = g (x) +^gi(x)ui, 

where we assume that all vector fields (that is, vector functions) §i are of class 
C°° . Consider the vector space spanned by the set of all functions of the type 

L 9i 1 --- L g ik h j( x ) ( 6 - 16 ) 
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over all possible sequences ii,...,ik, k > 0, out of {0, . . . ,m} and all j = 
1, . . . ,p, where L g a = Va.j for any function a and any vector field g. This 
is called the observation space O associated to the system. We say that two 
states X\ and Xi are separated by O if there exists some a G O such that 
a(xi) 7^ a(x2)- One can prove that if two states are separated by O then they 
are distinguishable. A sketch of the argument is as follows. Assume that X\ 
is indistinguishable from x-i and consider a piecewise constant control which 
is equal to u 1 on [0,ti), equal to v? on [ii,ii + £2); • • • , and equal to u k on 
\t\ + . . . + ifc-i, t\ + . . . + ifc). For small enough ij's this control is admissible for 
both Xi and x%, and by indistinguishability we know that the resulting output at 
time t = t\ + . . . + tk is equal for both. In general, we denote the Jth coordinate 
of this output value by 

h j (h,t2,---,tk,u 1 ,u 2 ,... ,u k ,x) (6-17) 

if the initial state is x. It follows that the derivatives with respect to the U's 
of this output are also equal, for x\ and X2, for every such piecewise constant 
control. One may prove by induction that 



k 



dti... dt k 



hj (ti,t 2 , ■ ■ ■ ,tk,u x ,u 2 , . . . ,u k ) 

t 1= t 2 =...=o 

= L Xl Lx 2 ...L Xk hj(x) 



where Xi(x) = go{x) + X^i u i9i{ x )- This expression is a multilinear function of 
the m''s, and a further derivation with respect to these control value coordinates 
shows that the generators in (6.16) must coincide at x\ and xi. In the analytic 
case, separability by O is necessary as well as sufficient, because (6.17) can be 
expressed as a power series in terms of the generators (6.16). 

The observability rank condition at a state io € 1 is the condition that the 
dimension of the span of 

{VL 9ii . . . L gik hj{x ) I h,...,ik e {0,...,m},j = l,...p) 

be n. An application of the Implicit Function Theorem shows that this is 
sufficient for the distinguishability of states near xq. For more details, see, for 
instance, [185], [199], and [311]. □ 

Exercise 6.4.3 Give an example of a system of class C°° with X = U = y = R 
which is observable but for which O does not separate. □ 

Exercise 6.4.4 Let x = f(x), y = h(x) be a smooth continuous-time system 
with no controls. Assume that /(0) = and /i(0) = 0. Relate the observability 
rank condition to the observability of the linearization about (0,0). □ 
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6.5 Realization Theory for Linear Systems 

Motivated by the discussion in Sections 3.3 and 6.2, it becomes of interest to 
study triples of matrices (A, B, C) over a field K, 

AeK nxn , BeK nxm , CeP x ", 

corresponding to linear time-invariant systems with outputs, and it is unam- 
biguous to call such a triple "controllable" or "observable," meaning that the 
corresponding discrete-time and (for K = M. or C) continuous-time systems have 
that property. The dimension of the triple is the number n, and m and p are 
the number of inputs and outputs, respectively. We denote by 

the set of all such (A, B, C)'s, and we use 

nc.o 

for the set of controllable and observable ones. Note that for IK = M. or C, S^° m _ 
is an open dense subset of S n , m ,p, in the topology for 



obtained when identifying this space with j£ n (™+ m +p) 

Assume that (A, B, C) is so that rank 0(A, C) = r and pick a decomposition 
as in Equation (6.8), where T G GL(n). Write 

D:=T- X B- ' Bl 



B 2 

and consider the pair (A\,B\). If this is not controllable, we can decompose it 
via Lemma 3.3.3. Let Si G GL(r) be so that 

Sr^Si=(^ A ^) and Sr^i^ 

and pick S := I € GL(n). Letting Q := TS, the following useful 

decomposition results: 

Lemma 6.5.1 For any triple (A, B, C) there exists some Q G GL(n) so that 

/A n A 12 
Q- 1 AQ= A 22 
\ A 31 A 32 A 33 , 

as well as 

Q- 1 B=\ j and CQ={Cn C 12 0) 
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for some matrices An, . . ., and the triple 

(Aii)-Bii,Cn) 

is controllable and observable. 

Proof. The pair (An,Bn) is controllable by construction, so it only remains 
to establish observability. Since 

^((l) 1 All) ,{Cl1 C i2)) = (O^n.Cn) *), 

one only needs to show that the matrix on the left has rank r. By construction, 
however, the pair there is similar to the pair (A\, C\), which was already known 
to be observable, so the result follows. I 

Lemma/Exercise 6.5.2 With the notations in Lemma 6.5.1, 

CA l B = CuA^Bn 

for alU = 0,1,2,... . □ 

Definition 6.5.3 The triple (A, B, C) is canonical if it is controllable and 
observable. □ 

For instance, the triple (An, Bn, Cn) in the above construction is canonical. 
One also calls a finite dimensional time-invariant discrete-time or continuous- 
time system E a canonical system if the corresponding triple is. 

A sequence 

A = {Ai,i = l,2,...}, 

where At £ K pxm , % = 1, 2, . . . for some fixed p and to, is a Markov sequence, 
and the Ai's are its Markov parameters. From Lemmas 2.4.6 and 2.7.13, the 
problem of realizing a discrete-time time-invariant linear behavior or a continu- 
ous-time time-invariant integral behavior is equivalent to the algebraic problem 
of factoring a Markov sequence as in (2.18), that is 

Ai = CA l - l B for all i >0. 

We say that a triple (A, B, C) for which this factorization holds is a realization 
of A; if there is a realization, A is realizable. 
From Lemmas 6.5.1 and 6.5.2 it follows that: 

Corollary 6.5.4 The Markov sequence A is realizable if and only if there exists 
a canonical triple realizing it. □ 

Thus, for any (continuous-time or discrete-time finite dimensional) time-in- 
variant linear system E, there exists a canonical system E c having the same 
behavior as E. Next we study when a given sequence A is realizable, and we 
show that canonical realizations are essentially unique. 
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Realizability 

Given a Markov sequence A and any pair of positive integers s,t, the (s,£)-th 
block Hankel Matrix associated to A is the matrix over IK: 



n a ,t(A) ■-- 



I A x A 2 ■■■ A t \ 

1 A 2 A 3 ■■■ At+i ' 



\A S A s +i ■■■ A s +t-i/ 



(6.18) 



of size ps x mt made up of st blocks, whose i,jth block is A4+J-1. 

For any triple (A, B, C) and each k we consider the matrices 0& as in equa- 
tion (6.6) as well as the reachability matrices 

R k (A,B) = [B,AB,...,A k ~ 1 B] 

(so that R„ = R). The main relations between these matrices and Hankel 
matrices are as follows: 

Lemma 6.5.5 The triple (A, B, C) realizes A if and only if 

O s (A,C)K t (A,B) = H a ,t(A) (6.19) 

for all s and t. □ 

Corollary 6.5.6 If (A, B, C) realizes A, then 

rankH St t{A) < max{rankO s (A, C), rankR t (A, B)} < n 

for all s, t. □ 

Note that if the sequence A is realizable by the triple (A, B, C), then also 

O s (A,C)AR t (A,B) = Wj +M> (6.20) 

the submatrix of H a +i,t formed by dropping the first block row. 

A triple is canonical precisely if both 0(A, C) = O n (A,C) and R(A,B) 
have rank n. In that case, there exist two matrices 0#(A,C) and R*(j4, B) 
such that 

0*{A 1 C)0{A,C) = H{A,B)H*{A,B) = I. (6.21) 

We use the notation "#" to emphasize that in the particular case of K = K 
or C one could use the respective pseudoinverses, though many other one-sided 
inverses exist. 

Corollary 6.5.7 If (A, B, C) realizes A and (A, B, C) is a canonical triple, then 

rank7-^,5 it („4) = n 

whenever s, t > n. 
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Proof. Using one-sided inverses satisfying (6.21), 0(A,C)R(A,B) = H n .n{A) 
implies that 

I = 0*(A,C)n n , n (A)R,*(A,B) 1 

so Hn,n{A) must have rank > n, and the conclusion follows. I 

Definition 6.5.8 Let (A,B,C) and (A,B,C) be two triples in S n _ miP . Then 
(A,B,C) is similar to (A,B,C), denoted 

(A,B,C)~(A,B,C) 

»/ 

T~ l AT = A, T~ 1 B = B, and CT = C 

for some T € GL(n). □ 

This is an equivalence relation, and the Markov sequences realized by any 
two similar triples are the same. The main result given below implies that, 
under a minimality assumption, the converse also holds; namely, if two minimal 
triples realize the same Markov sequence, then they must be similar. 

Equivalence corresponds to a change of variables in the state space. Note 
that Lemma 6.5.1 provides a particularly important decomposition under simi- 
larity. 

Definition 6.5.9 The n- dimensional triple (A,B,C) is minimal if any other 
triple (A, B, C) realizing the same Markov sequence A must have dimension at 
least n. □ 

Minimality is in principle hard to check, since it involves comparisons with all 
other possible realizations of the same Markov sequence. The next result shows 
that this property is equivalent to being canonical and hence can be checked 
directly in terms of the data describing the system; we use the statement to 
summarize all relevant properties of realizations. 

Theorem 27 Assume that A is a Markov sequence. Then the following prop- 
erties hold: 

1. If there is a realization of A, then there is also a canonical realization. 

2. A realization of A is minimal if and only if it is canonical. 

3. Any two minimal realizations of A must be similar. 

Proof. Statement 1 was given earlier as Corollary 6.5.4. We now prove state- 
ment 2. 

Let (A, B, C) be a canonical triple of dimension n. By Corollary 6.5.7, 
rank7^„. n („4) = n. It follows from Corollary 6.5.6 that every other realization 
has dimension at least n, so the triple is minimal. 
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On the other hand, if (A,B,C) is minimal, then it must be canonical, since 
otherwise from Lemma 6.5.2 and the construction in Lemma 6.5.1 it would 
follow that the triple (An, Bn, Cn) has lower dimension and realizes the same 
Markov sequence. 

Next we establish statement 3. Assume that (A,B,C) and (A,B,C) both 
realize A and are minimal, so in particular they must have the same dimension. 
Denote 

R = R(A, B) , R = R(A, B) , O = 0(A, C) , 6 = 0(A, C) 

and note that 

OR = OR and OAR = OAR 

because of Equations (6.19) and (6.20) applied with s = t = n. Let R^, . . ., be 
one-sided inverses as in Equation (6.21), for each of the triples. Then with 

T := RR # = (6 # 



it follows that T 1 AT = A. Applying Equation (6.19) with s = n,t = 1, and 
observing that Ri = -B,Ri = B, we also obtain that T~ l B = B. Finally, 
applying Equation (6.19) with s = l,t = n we obtain the remaining equality 
CT=C. I 

In terms of realizations of discrete-time time-invariant linear behaviors and 
continuous-time time-invariant integral behaviors, the Theorem asserts that 
canonical realizations exist if the behavior is realizable, and they are unique 
up to a change of basis in the state space. 

Remark 6.5.10 If two controllable triples are similar, then the similarity must 
be given by the formulas obtained in the proof of Theorem 27, so in particular 
similarities between canonical systems are unique. This is because if T is as in 
Definition 6.5.8, then necessarily 

T- l A l B = A l B 

for all i, so it must hold that T _1 R = R, and therefore 

T = RR # . 

In particular, this means that the only similarity between a canonical system 
and itself is the identity. In the terminology of group theory, this says that the 
action of GL(n) on triples is free. □ 

Theorem 27 leaves open the question of deciding when a realization exists; 
this is addressed next. 

Definition 6.5.11 The rank of the Markov sequence A is 

sup rank 7^ s ,t{A) . □ 

s,t 
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Remark 6.5.12 In terms of the infinite Hankel matrix which is expressed in 
block form as 



H(A) 



(M A 2 

A 2 A 3 



•A-s *^Vs+l 



V 



A t • • • ^ 

A t+ i ■ ■ 



A s +t-i 



J 



one may restate the definition of rank as follows. 

The rank of an infinite matrix such as H(A) is, by definition, the dimension 
of the column space of 7i(A), which is seen as a subspace of the space K°° 
consisting of infinite column vectors (xi,X2,Xs, . . .)' with entries over EC, with 
pointwise operations. 

When this rank is finite and less than n, all columns are linear combinations 
of at most n — 1 columns, and therefore all submatrices of 7i(A) must have 
rank less than n; this implies that A must have rank less than n. Conversely, 
we claim that if A has rank less than n, then the rank of the infinite matrix 
7Y(^4) is less than n. If this were not to be the case, there would be a set of 
n independent columns ci, . . . , c n of 7i(A). If so, let M, be the i x n matrix 
obtained by truncating each column Cj at the first i rows. Then some Mi has 
rank n: Consider the nonincreasing sequence of subspaces 



Qi 



ker Mi C 



and let Q := C\Qi] by dimensionality, there is some k so that Qk = Q, and if 
ieQ then Mix = for all i means that x is a linear relation between the Cj's, 
so x = 0. This means that Qk = and so M/~ has rank n as desired. Let s, t be 
such that Mfe is a submatrix of 7i s t . Then 



rank„4 > rank7^ st > rankM^ = n , 

contradicting rank„4 < n. 

The conclusion is that the rank of A is equal to the rank (possibly infinite) 
of H(A). □ 

Theorem 28 The Markov sequence A is realizable if and only if it has finite 
rank n. In addition, if this holds, then: 

(a) There is a canonical realization of dimension n. 

(b) ra,nkH n , n = n. 

Proof. If there is a realization of dimension k, then it follows from Corollary 
6.5.6 that A has rank at most k, and in particular the necessity statement is 
obtained. We show next that if A has rank n, then there is some rank n canonical 
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realization; this will establish property (a) as well as sufficiency Property (b) 
is then a consequence of Corollary 6.5.7. 

For the construction of the canonical realization, it is convenient to first 
generalize the notion of realizability to allow for infinite dimensional triples. 
In general, we consider objects (X, A,B,C) consisting of a vector space X and 
linear maps A : X — > X, B : K m — » X, and C : X — > K p , and say that A is 
realizable by (X, A, B,C) if 

C o A 1 - 1 o B = Ai 

for all i. These objects can be identified to discrete-time time-invariant linear 
systems; when X is finite dimensional, choosing a basis on X provides a triple 
realizing A. 

Now given any Markov sequence A, we let Xo denote the space K°° intro- 
duced in Remark 6.5.12. Note that the shift operator 

/*1\ /X2\ 



X 2 
%3 



X 3 

X 4 



V ; 7 V; 7 

is linear on Xo- Let A be defined as a p , the shift by p positions. Let B : K m — ► 
Xo be defined on the natural basis as follows: 

Be,j := jth column of 7"^(*4) 

and let C : Xo — > K p be the projection on the first p coordinates. 

We claim that (Xo, A, B, C) is a realization of A. Consider any i > and any 
j = 1, . . . ,m. Then CA l Bej is the vector consisting of the entries in positions 
ip + 1, . . . , ip + p of the jth column of Ti(A), that is, the jth column of A4+1, 
as wanted. 

The abstract realization just obtained is observable, in the sense that 



f]kevCA l = 0, 



since for each i the elements of ker C A % are precisely those vectors whose entries 
in positions ip + 1, . . . , ip + p all vanish. The desired canonical realization will 
be obtained by restricting to an A- invariant subspace of Xo; this restriction will 
also be observable, since the kernels of the restrictions still must intersect at 
zero. 

Let X be the subspace of Xo spanned by all the iterates A l Bej, over all i,j. 
This is ^4-invariant by definition; we denote the restriction of A to X again as 
A. Moreover, X contains the image of B, so we may consider B as a map from 
K m into X; similarly, we restrict C to X and denote it also by C. We claim 
next that this space equals the column space of H(A). Here the Hankel pattern 
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becomes essential: Every column of T~t(A) is of the form A 1 Bcj for some i and 
j; namely, the im + jth column (1 < j < m) is the same as A l Bej, for each 
i > 0. 

We conclude that (X, A, B, C) realizes A, where X is the column space of 
Ti(A) and therefore is n-dimensional. I 

The criterion in Theorem 28 is of interest because it allows us to check 
realizability, in the sense that one may look at Hankel matrices of increasing 
size and if a realization exists it will eventually be found. If a realization does 
not exist, however, there is no way of knowing so by means of this technique. 

If one is interested merely in investigating the existence of realizations, as 
opposed to studying minimality, the problem can be reduced to the scalar case 
(m = p = 1), which in turn belongs to the classical theory of linear difference 
equations: 

Exercise 6.5.13 Given any Markov sequence A, introduce the pm sequences 
corresponding to each coordinate, that is, 

J\ . — yj\.\)ij, \J^2)iji \'s\-3)iji • • • 

for each i = 1, . . . ,p and j = 1, . . . ,m. Show, not using any of the results just 
developed, that A is realizable if and only if each A %1 is. □ 

Exercise 6.5.14 Calculate a canonical realization, and separately calculate the 
rank of the Hankel matrix, for each of these examples with m = p = 1: 

1. The sequence of natural numbers 1, 2, 3, . . . . 

2. The Fibonacci sequence 0, 1, 1, 2, 3, 5, 8, ... . □ 

6.6 Recursion and Partial Realization 

An alternative characterization of realizability is through the concept of recur- 
sive sequences. We shall say that A is a recursive Markov sequence if there exist 
a positive integer n and scalars ai, . . . , a n such that 

Ak+n+l = Oi n Ak+n + Cln-lAk+n-1 + ■ ■ ■ + Ci2Ak+2 + CilAk+1 (6.22) 

for all fc > 0; in this case A is said to satisfy a recursion of order n. 

Proposition 6.6.1 The Markov sequence A is realizable if and only if it is 
recursive. 

Proof. If (A, B, C) realizes A, let 

X A (s) = s n - a n s n ~ l - . . . - ai 
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so that by the Cayley-Hamilton Theorem 

A n = a n A n - 1 + ... + aJ. 

Multiplying this last equation by CA k on the left and by B on the right, the 
recursion (6.22) results. Conversely, if there is a recursion of order n, all columns 
of the infinite Hankel matrix must be linear combinations of the first nm columns 
(that is, the columns appearing in the first n blocks), so the matrix has finite 
rank and therefore A is realizable. I 

Corollary 6.6.2 If A is a Markov sequence of finite rank n, then A = if and 
only if 

Proof. By Theorem 28 there exists a realization of dimension n, and therefore, 
by the proof of the Proposition, there is a recursion of order n. Recursively, all 
A l = 0. I 

Corollary 6.6.3 If A 1 and A 2 are two Markov sequences of finite ranks n\ and 
7i2, respectively, then A 1 = A 2 if and only if 

A\ = A 2 , i = 1,. . . ,n 1 + n 2 - 

Proof. The sequence A := A 1 — A 2 = {A} — A 2 } has rank at most n := ni + ri2, 
as follows by considering its realization: 

A o i) (2) ^ -*•> 

where (A,, Bi,Ci) is a realization of A 1 of dimension rij. Then Corollary 6.6.2 
gives the result. I 

Thus, sequences of rank < n are uniquely determined by their first In 
Markov parameters. 

There is a direct construction of a realization from the recursion (6.22) and 
the Markov sequence. This is as follows. Let 7"^ „ be the matrix obtained by 
dropping the first k blocks of rows from Ti n+ j c ,n{^), that is, the matrix 

■Ak+i ■ ■ ■ ■Ak+n 



,-4fc+n • • • Ak+2n-l 

including the case fc = 0, where this is just H nn {A). Then (6.22) implies 

A ^ k n+k,n = n n Vk+l,n ( 6 - 23 ) 
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for all k > 0, where 



( ° 




I 



. 

1 . 






\ot\I 




O-lI 


. 
a 3 J . 


I 

• a n I / 



(6.24) 



analogously to the controller form in Definition 5.1.5, except that this is now a 
block matrix. Each block has size p x p, and the matrix A is of size np x np. 
Equation (6.23) implies that 



■"■ i~i-n,n\'A-) — "n+i,ri 



for all i > 0, and so 

(/ 

for all i > 0. Letting 
and 

B 



Q)A*H n JA) 



C 



(I 



O 



Vo7 





■A-i+l 



'^n,n\^) 



(l\ 



\oJ 



f A A 



A 



\A n J 



(6.25) 



(6.26) 



(6.27) 



there results a realization of A of dimension np. Since O n (A, C) = I: 

Lemma 6.6.4 The system (A,B,C) given by (6.24), (6.27), and (6.26) is an 
observable realization of A. □ 

This is often called an observability form realization of A. 

Remark 6.6.5 Let A be any recursive Markov sequence. Consider the trans- 
posed sequence 

J\ := y^i, y\.2i ■ ■ ■ ■ 
This satisfies a recursion with the same coefficients a^S. For these coefficients we 
let (A, B, C) be the observability form realization of A'. Since CA l ~ l B = A\ for 
all i, also B'(A'y~ 1 C' = Ai for all i, and the system (A 1 , C', B') is a controllable 
realization of A. We have obtained the controllability form realization of 
A: 



/0 
/ 
/ 

Vo 



ai/\ 

ail 

a 3 I 

1 a n I J 



B 







Vo/ 



C = (Ai A 2 



•A-n ) 
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of dimension mn. □ 

Corollary 6.6.6 If p = 1 or m = 1, the minimal possible order of a recursion 
equals the rank of A. Furthermore, there is a unique such minimal order recur- 
sion, and its coefficients are those of the (common) characteristic polynomial of 
canonical realizations of A. 

Proof. The proof of Proposition 6.6.1 shows that, for arbitary m,p, if there is a 
realization of dimension n, then there is a recursion of order n. Conversely, in the 
cases p = 1 or m = 1 there is always a realization of dimension equal to the order 
of any given recursion, namely the observability form or the controllability form 
realization, respectively. The second assertion follows from the constructions. I 

Remark 6.6.7 Observability form realizations are in general not controllable, 
except if p = 1. In this case, if (ai, . . . , a n ) give a minimal recursion then by 
Corollary 6.6.6 this realization must be minimal. In fact, 

R n (A,B) = H n ,n , 

which has rank n. Of course, for p > 1 it is possible to reduce any such realiza- 
tion to a controllable one using the Kalman decomposition. □ 

Exercise 6.6.8 Show by providing a counterexample that the hypothesis that 
either p = 1 or m = 1 cannot be dropped in the first assertion on Corollary 
6.6.6. □ 

Given any finite sequence Ai,A2, ■ ■ ■ ,A r , this is always part of a realizable 
Markov sequence, since 

Ai,A 2 ,-.-,A r ,0,0,...,0,... 

has finite rank. More interesting is the following fact. Given any finite sequence 
A\ , A2 , • • • , A r , whenever s + t < r+l we can consider the Hankel matrices 7i s> t 
obtained using formula (6.18). 

Lemma 6.6.9 Assume that A\, A 2 , . . • , A2 n are 2n matrices in K pxm for which 

rankH n , n = rank7-^„ + i in = rank7i„ ]n+ i = n . (6.28) 

Then there exists a (unique) Markov sequence A of rank n whose first 2n Markov 
parameters are Ai,A2, ■ ■ ■ ,A2n- 

Proof. From the equality rank7i„„ = rank7i„ +1 „ it follows that the last p 
rows of H n +i^ n must be linear combinations of the rows of Ti n ,n- This means 
that there must exist p x p matrices 

C,- , i = 1, .... 71 
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such that 

Aj = CiAj-i + . . . + C n Aj- n (6.29) 

for each j = n + 1, . . . , 2n. Similarly, rank7i„ n = rank7i„ n+1 implies that the 
last m columns of Ti-n.n+i must be linear combinations of the columns of 7i n>ra . 
So there must also exist m x m matrices 



such that 



Di , i = l,...,n 



Aj = Aj-iD-l + . . . + Aj_ n D n (6.30) 



for each j = n + 1, . . . , In. 

We now define Aj for j > 2n recursively using the formula (6.29), and let 
A be the Markov sequence so obtained. It follows from this definition that all 
rows of 7~t{A) are linearly dependent on the first pn rows. So A has rank at 
most pn; we next show that its rank is in fact just n. For this, it is enough 
to establish that (6.30) holds for all j > In as well, since this will then imply 
that all columns depend linearly on the first nm columns, and therefore that 
the rank of Ti(A) is the same as the rank of 7i. n , n , which is n. By induction on 
j, 



Aj+i — 2_^ CiA 



j+i-i 




as desired. 

Uniqueness follows from the fact that any other extension would agree in its 
first In parameters and hence would have to be the same, because of Corollary 
6.6.3. I 

The previous result can be used to give an explicit description of the quotient 
space obtained when identifying triples up to similarity. Let M. n ,m.p denote the 
set 

{(.Ai, . . . ,A 2n ) | rank7i„,„ = TankTC n +i,n = rankH n , n +i = n} . 

Note that this can be thought of as a subset of K 2nm P ; an d as such it is a set 
defined by polynomial equalities and inequalities. Let 

R ■ Q c '° —a A/f 

y ■ ^n.m.p J vl n,m,p 
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be the "behavior" function 

(A,B,C) h-> {CB,...,CA 2n - l B). 

That P indeed maps into M. n ,m,p follows from Theorem 28 (p. 288). Moreover, 
from Lemma 6.6.9 we know that (3 is onto. From the uniqueness result, we also 
know that, if 

(3(A,B,C) = /3(A,B,C), 

then (A, B, C) and (A, B, C) are two canonical triples realizing the same Markov 
sequence and so, by Theorem 27 (p. 286), Part 3, that these two triples are 
similar. This discussion can be summarized as follows. 

Corollary 6.6.10 The map j3 induces a bijection between the quotient space 

S n°m,pl ~ and M n ,m,p- U 

Exercise 6.6.11 Let K = R or C. Let M n . m ,p have the topology induced 
by K 2 ™" 125 , and let S^° m „/ ~ be given the quotient topology, when S%° m „ is 
thought of as a subspace of j£™(™+P+ m ) Show that (3 induces a homeomorphism 
on the quotient space. {Hint: Establish first that the realization given in the 
proof of Theorem 28 can be made to be locally continuous on the sequence. This 
is done by observing that, by Cramer's rule after choosing a set of n linearly 
independent columns of 7i n ,n, the entries of the operator A can be taken to be 
rational on the coefficients of the Markov parameters. This provides a covering 
of M n ,m.p by open subsets V and continuous mappings p : V — » S%° m _ such 
that (3p is the identity.) □ 

Remark 6.6.12 (This remark uses some concepts from differential geometry.) 
In the case when K = R (or the complex case) one can show that the quotient 
space S^'° m „/ ~ has a differentiable manifold structure under which the natural 
projection is smooth. This fact can be established constructively, by exhibiting 
an explicit set of charts for this quotient manifold, or one can use a general 
theorem on group actions, the quicker path which we choose here. Consider the 
action of GL(n) on S^'° mp , 

T.{A,B,C):= {T^AT.T^B^T), 

seen as a smooth action of a Lie group on a manifold (S%° m p is an open subset of 
^n(n+m+p)y This ac ti n is free (see Remark 6.5.10). According to Proposition 
4.1.23 in [2], the quotient will have a differentiable manifold structure for which 
the quotient mapping S%° mp — » S%° mp l ~ is a smooth submersion, provided 
that the graph of the similarity relation is closed and that the action is proper. 
Moreover, in this case the natural map 

qc,o Q c i° I 

defines a principal fibre bundle (same reference, Exercise 4.1M). Properness of 
the action means that the following property must hold: Whenever {Sj} is a 
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convergent sequence of triples and {Xj} is a sequence of elements of GL(n) for 
which {Tj.Ej} is a convergent sequence of triples, the sequence {Tj} must have a 
convergent subsequence. So we must prove that the action is closed and proper. 
Assume that the sequences {Ej} and {Tj} are as in the above paragraph, 
and let E^ := Tj.E, for each i. By assumption, there are systems E and E' so 
that 

E, -> E and E^ -> E' . 

We use primes and subscripts to denote the matrices A, B, C for the various 
triples. The triples E and E' must realize the same Markov sequence, since this 
is true of the corresponding pairs (Ej,EQ and the sequence's elements depend 
continuously on the triple. Therefore the matrix 



R(R' 



<# 



provides a similarity between these two triples. Observe that R(j) (the n-step 
reachability matrix for the triple Ej) converges to R. Moreover, and this is the 
critical observation, one may also assume that (R', ••,)#, the one-sided inverse 
of the n-step reachability matrix for the triple E^, also converges to (R')#. 
The reason for this latter fact is that one may pick such a one-sided inverse 
continuously about any given system: Just use Cramer's rule after choosing 
a set of n linearly independent columns of R' (these columns remain linearly 
independent for triples near the triple E'). We conclude that 

T i = R (i) (R' ( . ) )# - T 

because of uniqueness of the similarity between two minimal systems (Remark 
6.5.10). This establishes properness. (In fact, we proved that the sequence Tj 
itself is convergent, rather than merely a subsequence.) 

The proof of closeness is even easier. We need to see that, if 

Ej ~ E^ for all i 

and 

Ej ► E , ij i ► E , 

then necessarily E and E' are similar. This is immediate from the uniqueness 
Theorem, because by continuity these two triples must give rise to the same 
Markov sequence. □ 

Corollary 6.6.10, Exercise 6.6.11, and Remark 6.6.12 provide only a very 
brief introduction to the topic of families of systems and more specifically to 
moduli problems for triples. Much more detailed results are known; For instance, 
^n°m pi ~ nas a natural structure of nonsingular algebraic variety consistent 
with the differentiable manifold structure given above, for K = R, C. 
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6.7 Rationality and Realizability 

We now characterize realizability of a Markov sequence in terms of the ratio- 
nality of an associated power series. 

Consider the set of all semi-infinite sequences 

Ofe,0/; + i, ... ,00,01,02,03, ... ,o;, .. . 

formed out of elements of K. The starting index k may be negative or positive 
but it is finite; however, the sequence may be infinite to the right. We think of 
these sequences as formal Laurent series in a variable s : 



E< 



,diS 

i—k 



where a, 6 K for each i. If the sequence is not identically zero and k is the 
smallest integer so that Ofc 7^ 0, then the series has order k. If k > 0, this is just 
a formal power series in the variable s _1 . If all coefficients Oj, i > vanish, this 
is simply a polynomial on s. Let ^((s^ 1 )) denote the set of all such Laurent 
series. 

The set /C((s -1 )) has a natural vector space structure over K, corresponding 
to coefficientwise operations: 



(E 



a , s 



£• 



V X>* _< ) := ^(ai + fci)* - ', (6.31) 

but the use of the power series notation also suggests the convolution product 

£>*-') := £>-', ( 6 - 32 ) 

where for each Z € Z 

C( := ]P aibj 

is a finite sum because there are at most finitely many nonzero coefficients with 
negative i and j. With these operations, /C((s -1 )) forms a ring. It is an integral 
domain, that is, the product of two nonzero sequences cannot be zero, because if 
k\ and ki are the orders of these sequences, then the product has order k\ + k^- 
(In fact, since IK is a field, it can be shown that /C((s -1 )) is also a field, but this 
is not needed in what follows.) 

More generally we consider series whose coefficients are matrices of a fixed 

size: 

00 



Y,A t s- 



l = k 



where Ai G K pxm for all i. The sum and product given by formulas (6.31) and 
(6.32) are still well defined (assuming that sizes match), where now each Ai, 
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Bj and Ci is a matrix. The usual distributivity and associativity properties of 
matrix product hold. We let tC pxm ((s~ 1 )) denote the set of all matrix Laurent 
series with fixed p and m. 

A series W(s) s X7 ,xm ((s~ 1 )) will be said to be rational if there exist a 
monic polynomial 

q(s) = s n -a n s n - 1 - ...- ai (6.33) 

and a matrix polynomial 

P{s) = P + Pis + ... + P h s h (6.34) 

in KP xm [s] such that 

qW = P 

(which is also written as W = q P, or P/q). For instance, any series having 
only finitely many terms is rational, since 

A k s k + ... + A + A lS ~ l + ... + A lS ~ l 
can be written as 

\{A k s k+l + ... + A s l + A^ 1 - 1 + . . . + A t ) . 
For another example consider the scalar series 

1 + 5 _1 + S" 2 + ... + s~ k + ... , 

which is rational since it equals s/(s — 1). 

We associate to each Markov sequence A = (Ai, -4.2>-4.3) • • •) its generating 
series 

oo 

In terms of this we may state another criterion for realiz ability: 

Proposition 6.7.1 A Markov sequence is realizable if and only if its generating 
series is rational. 

Proof. Because of Proposition 6.6.1, we must simply show that W A is rational 
if and only if A is recursive. 

Note first that, in general, if W = q~ x P is rational and has order > 1 and 
q is as in (6.33), then the polynomial P in (6.34) can be taken to be of degree 
at most n — 1, since all terms in qW corresponding to s h , h > n, must vanish. 
So rationality of W A is equivalent to the existence of elements ai, . . . , a n in K 
and matrices P<y, . . . , P n -\ over IK such that 

(«» - On*"" 1 - . . . - oi) (f^A t s-A =p + p lS+ ... + p n _ lS ™- 1 . (6.35) 
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If such an equation holds, then comparing coefficients of s^ k ^ 1 results in 

Ak+n+l = Ol n Ak+n + Oln-l-A.k+n-1 + ■ • • + a 2 Ak+2 + Ql-Afc+l 

for each k > 0, so the sequence is recursive. Conversely, if this equation holds 
one may just define the matrices P 4 by 

n-j-l 

Pj := A n -j - ^ a i+J+1 Ai j = 0, . . . , n - 1 , (6.36) 

i.e., the equations that are derived from (6.35) by comparing the coefficients of 

M,...,*"- 1 . ■ 

The proof shows that the minimal degree of a possible denominator q equals 
the minimal order of a recursion satisfied by A. 

Corollary 6.7.2 If m = 1 or p = 1 and IU4 = P/q with q monic of degree 
equal to the rank of A, then q is the (common) characteristic polynomial of the 
canonical realizations of A. 

Proof. We know from Corollary 6.6.6 that for m or p equal to 1 the coefficients 
of a minimal recursion are those of the characteristic polynomial. The q in 
the statement must be a denominator of minimal degree, since a lower degree 
polynomial would give rise to a lower order recursion and therefore to a lower 
dimensional realization. From the above construction, this polynomial then 
corresponds to a minimal recursion. I 

Exercise 6.7.3 Show that, if m = p = 1 and if IU4 = P/q with P of degree 
< n — 1 and q of degree n, then A has rank n if and only if P and q are relatively 
prime. {Hint: Use Corollary 6.6.6 and the fact that /C((s -1 )) forms an integral 
domain.) □ 

Remark 6.7 .4 One can show directly that realizability implies rationality, as 
follows. Elements of IC pxm ((s~ 1 )) can be identified naturally with p x m ma- 
trices over the ring /C((s _1 )), and this identification preserves the convolution 
structure. Under this identification, rational elements are precisely those that 
correspond to matrices all whose entries are rational. In particular, the matrix 
(si — A)^ 1 , the inverse over /C((s~ 1 ))™ x ", is rational. On the other hand, it 
holds that 

oo 

(si- A)- 1 = ^2 A 1 - 1 s- 1 

4=1 

from which it follows that the series 

oo oo 

C(sI-A)- 1 B = ^ j CA i - 1 Bs- t = ^A lS - 1 

is rational if realizability holds. The minimal dimension of a realization is also 
called the McMillan degree of the corresponding rational matrix. □ 
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Remark 6.7.5 When K = M. or C, one may use complex variables techniques 
in order to study realizability. Take for simplicity the case m = p = 1 (otherwise 
one argues with each entry). If we can write Wj^{s) = P(s)/q(s) with degP < 
deg q, pick any positive real number A that is greater than the magnitudes of all 
zeros of q. Then, Wj^ must be the Laurent expansion of the rational function 
P/q on the annulus \s\ > A. (This can be proved as follows: The formal equality 
qWa = P implies that the Taylor series of P/q about s = oo equals Wj^, and 
the coefficients of this Taylor series are those of the Laurent series on \s\ > A. 
Equivalently, one could substitute z := 1/s and let 

q(z) := z d q(l/z) , P(z) := z d P(l/z) , 

with d := degg(s); there results the equality q(z)W(l/z) = P(z) of power 
series, with q(0) ^ 0, and this implies that W(l/z) is the Taylor series of P/q 
about 0. Observe that on any other annulus Ai < \s\ < A 2 where q has no 
roots the Laurent expansion will in general have terms in s k , with k > 0, and 
will therefore be different from W^.) Thus, if there is any function g which 
is analytic on \s\ > /i for some /x and is so that Wj^ is its Laurent expansion 
about s = 00, realizability of A implies that g must be rational, since the Taylor 
expansion at infinity uniquely determines the function. Arguing in this manner 
it is easy to construct examples of nonrealizable Markov sequences. For instance, 

1 1 1 
A = 1 - — — 

'2' 3!' 4!'"" 

is unrealizable, since W.4 = e x ' s — 1 on s 7^ 0. As another example, the sequence 

1 1 1 
A = 1 - - - 

'2'3'4'"' 

cannot be realized because W.4 = — ln(l — s _1 ) on \s\ > 1. □ 

Input/Output Equations 

Rationality and realizability can also be interpreted in terms of high-order dif- 
ferential or difference equations satisfied by input/output pairs corresponding 
to a given behavior. 

The input/output pairs of the behavior A are by definition the possible pairs 
(w, X(u>)). For each u € VS a,T ' in the domain of the behavior, this is a pair 
consisting of a function in VS ^' and one in ty a ' T K 

Assume from now on that A is a time-invariant continuous-time integral 
behavior with analytic kernel 

00 f, 

K(t) = E-^+iry ( 6 - 37 ) 



fe=0 



(assume K is entire, that is, the series converges for all t), where A = Ai, A2, ■ ■ ■ 
is a Markov sequence over K = K or C. Observe that whenever lu is r — 1-times 
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(continuously) differentiable the corresponding output function X(u>) is r-times 
differentiable; such a pair {ui, A(u>)) is of class C r . 
The behavior A is said to satisfy the i/o equation 

n — \ n— 1 

!/<*>(*) = ^> i+1 y«(*) + £^« (i) (*)> (° 8 ) 

where ai, . . . , a n £ K and Pq, . . . , P„_i £ K pxm , if this equation is satisfied 
by every i/o pair u = u),y = A(w) of A of class C n , for all i G [ cr j T )- The 
nonnegative integer n is called the order of the i/o equation. (For n = the 
equation is j/(i) = 0.) 

It is a basic fact in linear system theory that the existence of an i/o equation 
is equivalent to rationality and hence to realizability by a finite dimensional 
linear time- invariant system. 

Proposition 6.7.6 The behavior A satisfies (6.38) if and only if W_a = q 1 P, 
where q and P are as in Equations (6.33) and (6.34), with the same a^'s and 
Pi's. 

Proof. We first make some general observations. If K is as in (6.37), then 

K^(t) = f> fc+r+ ijb 

fc=0 

for each r = 0, 1, .... If, in addition, A is known to satisfy the order-n recursion 



Ak+n+i = y^ djAk+j (6.39) 



for all k > 0, then 



oo n k 

K {n \t) = £I>^+4 

fe=0j=l 
n oo fc 

j = l k=0 
n 

i=i 

for all t. 

On the other hand, for each i/o pair (u>,rj) of class C" _1 one concludes by 
induction from 

rj(t) = / K(t — /i)cij(/i) <i/i 
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that 



?=1 
for each r = 0, . . . , n. In particular 



v v(t) = Y^ A ^ (M) (t) + / K {r \t - n)u{n) d/i (6.40) 



n {n) 



n n ,. t 

(t) = Y.^^it) + Y, a i KV-V(t-n)w(jt)dn (6.41) 



*=1 J = l 

if .4 satisfies the above recursion. Since also 

n — 1 n — 1 



^a m ryW(t) = ^ a,+i 



i=0 j=0 

it follows that when a recursion exists 

n-l 



Y J Aiu {l - l) {t)+ KM{t-p)w{n)dn 



i=0 

n n— 1 



z=0 
n 

j=l j=0 1=1 

n-l I n-j-1 \ 

= ^2[-An-j- Y, "i+i+iA w°'»(f) (6.42) 

j = V 8 = 1 / 

for all t. 

We conclude from Equation (6.42) that if qWj± = P, which implies that both 
Equations (6.39) and (6.36) hold, then also the i/o equation (6.38) is valid. 

Conversely, assume that (6.38) is true for all i/o pairs of class C™^ 1 . Pick 
any arbitrary vectors Vq, . . . , u n _i € K m , and any interval [<j, r), and for these 
let lu be the control on [a, r) defined by 

i=0 

Using again (6.40), the i/o equation at time t = a gives that 

n — 1 n—1 

AiV n ^i + . . . + A n V = Y <Xi+l(AiVi-i + ... + AiV a ) + Y P i V i 

i=0 j=0 

must hold. Since these vectors were arbitrary, one may compare coefficients and 
equation (6.36) results. It also follows by differentiating (6.38) that 

n— 1 n—1 

y (n + k + i) {t) = £ ai+1J ,(*+*+i)(t) + Y,PiU {i+k+1) {t) 

i=0 i=0 
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for all k > and pairs of order C n+ . Applied in particular to the constant 
controls lu = vq, vq arbitrary, and evaluating at t = a, the recursion (6.39) 
results, and we know then that qW^ = P, as desired. I 

A similar result holds for discrete-time systems. A linear time-invariant 
discrete-time behavior is said to satisfy the i/o equation of order n 

n—1 n—1 

y{t + n) = J^ Oi+iJ/(* + i) + J2 p Mt + (6-43) 

4 = 4 = 

if this equation holds for each i/o pair with u> € W- a ' T > and each a < t < r — n. 

Lemma/Exercise 6.7.7 The discrete-time time-invariant behavior A satisfies 
(6.43) if and only if W^ = q~ x P, where q and P are as in Equations (6.33) and 
(6.34), with the same a»'s and Pj's. □ 

Exercise 6.7.8 Refer to Exercise 3.2.12. (Take all constants equal to one, for 
simplicity.) 

(a) Using y = h as the output, find an i/o equation of order 4 and the transfer 
function of the input/output behavior of E. 

(b) Repeat the computation taking instead x 3 = as the output; why is a 
transfer function with denominator of degree two obtained this time? Find 
a two-dimensional realization of the new i/o behavior. □ 

Exercise 6.7.9 Refer to Exercise 3.2.13, and take for simplicity M = m = F = 
9 = 1=1- 

(a) Using y = x\ = 5 as the output, find an i/o equation of order 4 and the 
transfer function of the input /output behavior of E. 

(b) Repeat the computation taking instead x% = <j> as the output; show that 
now there is an i/o equation of order three. Find a three-dimensional 
realization of this i/o behavior. □ 

6.8 Abstract Realization Theory* 

There is an abstract theory of realization, which is of interest especially in the 
case of realizability by finite systems. We restrict attention here to complete 
time-invariant behaviors; unless otherwise stated, throughout this section A 
denotes a fixed such behavior. Accordingly, we wish to study realizations of A by 
time-invariant initialized complete systems (E,x ). The realizability condition 
is that 



* This section can be skipped with no loss of continuity. 
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for all t € T + and each to £ ll' '*-'. We drop the superscripts 0, t and/or the 
subscripts £ and x° on the left when they are clear from the context. 

The first observation is that, at this level of abstraction, every behavior is 
realizable. To see this, consider the set 

n := |J u[°< T > 

T£T + 

and the map 

<f>(i(T,<7,v,ui) := VLO T ~ a 

corresponding to concatenation. That is, if 

W 6Ul ff ' T » and ueU^ T \ 

then this is the control equal to v on [0,T) and equal to 

cu(t-T + a) 

if te [T,T + r-a). 

Lemma/Exercise 6.8.1 The data (T, fi,U, 4>) define a complete time-invari- 
ant system. □ 

We add the output function 

h\ : — » y : lo i— > A(w) 

and the initial state a; = o. Since i/>n(i, 0, o, w) = w for all u> defined on intervals 
of the form [0,i), it follows that 

h\(<f>n{t,0,o,w)) = h\{uj) = A(w) , 

and the following is established: 

Lemma 6.8.2 The initialized system with outputs 

Sn,A := (T, il,ll, 0n,y,/iA,o) 

is a realization of A. This system is complete, time-invariant, and reachable 
from the state x° = o. □ 

Of course, the size of the state space of this system is huge, since it merely 
memorizes all inputs. A much more interesting realization results when one 
identifies indistinguishable states. The construction needed is more general, 
and it applies to arbitrary complete time-invariant systems, as follows. 

Let E = (T, X, II, <f>, y, h) be any time-invariant complete system with out- 
puts. Consider the space 

X := X/ ~ 
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consisting of all equivalence classes under indistinguishability. For each a < r 
in T + , each equivalence class [x], and each cu e W- (T ' T > let 

4>(T,a,[x],aj) := [4>{t, a, x, w)] , 

that is, pick an arbitrary element in the equivalence class, apply the transitions 
of E, and then see in which equivalence class one lands. This map is well defined 
in the sense that the result is independent of the particular element [x\. In other 
words, for every i,z£lso that x ~ z and every u>, also x\ := 4>{t, <t, x,u>) and 
z\ := 4>{t, a, z, ui) are indistinguishable. This is clear from the fact that, if v is 
a control distinguishing x\ and Z\, then the concatenation iqv distinguishes x 
and z. Similarly, the map 

h([x]) := h(x) 

is well defined because indistinguishable states give rise in particular to identical 
instantaneous outputs. 

This allows us to define a system 

E := (T,X,U,j>,y,h) 

called the observable reduction of E. It satisfies that 

h(<f(T,a, [x],u)) = h([(/)(T, a, x, w)]) = h(cj)(T,a,x,uj)) 

for all lu and x, so in particular with initial state 

7> := [a: ] 

it realizes the same behavior as (T,,x°), for each fixed x° € X. It is indeed 
observable, since [x] ~ [z] implies 

h((j){T,a,x,ujj) = h(<f)(T,o-,[x],u))) = h(4>(T,a,[z],u))) = h(<f>(T,<j,z,u>)) 

for all u), and therefore [x] = [z\. Finally, note that, if the original system S is 
reachable from a state x°, then the observable reduction is reachable from [x ]. 
Summarizing: 

Lemma 6.8.3 For each system E and each x° € X: 

1. E is observable. 

2. If E is reachable from x° , then E is reachable from x°. 

3. The i/o behaviors of (E,x ) and (E,a; ) coincide. □ 

An initialized system (E, x°) is a canonical system if it is reachable from 
x° and observable. This terminology is consistent with that used for linear 
systems, when applied with initial state x a = 0, since reachability from the 
origin is equivalent to complete controllability in that case. 
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Given any behavior A, let £a be the observability reduction of the system 
£q,a in Lemma 6.8.2. This is again reachable (from o), and by construction 
is also observable (as well as time-invariant and complete). We conclude as 

follows: 

Theorem 29 The initialized system with outputs (£a,o) is a complete, time- 
invariant, canonical realization of X. □ 

Our next objective is to show that canonical realizations are unique up to 
a relabeling of states. Assume given two initialized systems (£j,x°). A system 
morphism 

T : (Ei,a;?)-(E 2 ,a;§) 

is by definition a map 

1 \ ~L\ — > X2 

such that Tx\ = x 2 , 

T{4>i{t,o-,x,uj)) = (f> 2 (T,o;T(x),u>) 

for all u G tl' CT ' T ' and all x £ X, and h(x) = h(T(x)) for all iGl It is a system 
isomorphism if there exists another system morphism S : (£2,^2) — * (Si; x i) 
such that the compositions T o S and S o T are the identity on X2 and Xi, 
respectively. Since at this set-theoretic level a map is invertible if and only if it 
is one-to-one and onto, it follows easily that T is an isomorphism if and only if 
the underlying map T : Xi — > X2 is bijective. 

Theorem 30 Let (Ei,a;5) and (£2,212) be two (complete, time-invariant) ini- 
tialized systems with output. Assume that £1 is reachable from x\, £2 is observ- 
able, and they realize the same behavior. Then there exists a system morphism 
T : {Yji,x\) — y (£2, #2). Furthermore: 

1. There is a unique such morphism. 

2. If T12 is reachable from x\, then T is onto. 

3. I/£i is observable, then T is one-to-one. 

4- If both systems are canonical, then T is an isomorphism. 

Proof. Consider the set G consisting of all pairs (xi,X2) £ li x 1 2 for which 

/h(<Mt,ct,3;i,u/)) = h 2 {(P 2 {T,cr,x 2 ,uj)) Vct < t,Vw e U [a - r) 

(indistinguishable but in different state spaces). We claim that G is the graph 
of a system morphism. 

First note that, if (ari, x 2 ) and (xi, x' 2 ) are both in G, then the definition of 
G forces x 2 and x' 2 to be indistinguishable, so by observability it follows that 
x 2 = x' 2 , which means that G is the graph of a partially defined map. Its domain 
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is all of Xi: by reachability, given any ij 6li there exists some control u/ so 
that 

<j)l(<T,0,Xi,U>') = X\ 

and we define x 2 ■= 4> 2 (a,0,x 2 ,io'); then for all lo £ W' 7 ' t \ 

hi((t)i(T,a,xi,u))) = \{oj'oj) = h 2 (4> 2 (T,<j,x 2 ,w)) , 

which shows that (xi,X 2 ) £ G. We have proved that G is the graph of a map 
1 '. JL\ — ► X2. 

To see that T defines a morphism, observe that (x^, x 2 ) £ G, by definition of 
G and the fact that the behaviors coincide, and also that for each (xi,x 2 ) £ G 
necessarily h\{xi) = h 2 (x 2 ) (from the case it = t in the definition of G). Finally, 
consider any lo £ w a ' T > and let for i = 1, 2 

Xi := 4>i(T,a,Xi,ui) 

so we need to show that (xi,X 2 ) £ G, too. This follows trivially from the 
definition of G and the semigroup property. 

For any morphism To, if (xi,x 2 ) is in the graph of To, then also 

(</>i(r, a, Xi,U)),<f>2(T, a, x 2 , w)) 

is in the graph, for all w; it follows that also 

hi(<pi(T,a,xi,w)) = h 2 ((f) 2 {T,a,x 2 ,uj)) 

for all lo, so (xi,x 2 ) £ G. This means that the graph of To is included in G, 
which implies T = T , establishing the uniqueness claim. 

Finally, if Y, 2 is reachable, then an argument totally analogous to that used to 
prove that T is defined everywhere shows that T must be onto, and an argument 
like the one used to show that T is single-valued gives that T is one-to-one if 
Si is observable. When both systems are canonical, the previous conclusions 
show that T must be bijective. Alternatively, the Theorem can be applied twice, 
resulting in a T : Si — > Y, 2 and an S : Y, 2 —> £1; the compositions ToS and SoT 
are system morphisms, and since the identities i" : £1 — » £1 and / : £2 — > £2 are 
morphisms, too, the uniqueness statements imply that the compositions equal 
the identity. I 

Thus, just as in the linear case, one may conclude that every behavior is 
realizable by a canonical system, and canonical realizations are unique. No 
finiteness statements have been made yet, however. 

Remark 6.8.4 One can in fact obtain many of the results for the linear theory 
as consequences of the above abstract considerations. For instance, take the 
proof of the fact that canonical realizations of linear behaviors must be unique 
up to a linear isomorphism. Assume that two systems are given, with the same 
spaces U, y , which are assumed to be vector spaces over a field IK, that the state 
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spaces Xi,X2 are also vector spaces, and that the maps fa and hi are linear in 
(x,u>) and x, respectively. Furthermore, we assume that the initial states are 
zero. Then we claim that the set G is a linear subspace, which implies that 
the unique morphism T must correspond to a linear map Xi — > X2. Indeed, if 
(xi, X2) and (z\, z 2 ) are in G and if k £ K, then linearity gives that 

h(fa(T,a,xi + kzi,oj)) = h(fa(T,a,xi,0)) + kh((j)i(T,a,zi,u;)) 

= h(fa(T, a, x 2 ,0)) + kh(fa>(T, a, z 2 ,u)) 
= h(fa(T, a, x 2 + kz 2 ,u>)) , 

which implies that (x\ + kzi,x 2 + kz 2 ) € G. In particular, if both systems are 
canonical, then G is the graph of a linear isomorphism. 

Arbitrary linear isomorphisms are not very interesting when considering in- 
finite dimensional linear systems. For example, in the context of systems whose 
state spaces X^ are Banach spaces, the output value set y is a normed space, 
and the maps fa and hi are assumed to be bounded (continuous) linear opera- 
tors, one may want to conclude that the morphism T given by the Theorem is 
bounded, too. This is an easy consequence of the above proof: It is only nec- 
essary to notice that G must be closed (because of the assumed continuities), 
so by the Closed Graph Theorem (see, for instance, [399], Theorem 4.2-1) the 
operator T must indeed be continuous. If both systems are canonical, T _1 is 
also bounded, by the same argument. □ 

We now turn to minimality. A minimal system will be one for which 

card X = n < 00 

and with the property that any other system realizing the same behavior must 
have a state space of cardinality at least n. 

Lemma 6.8.5 A (time-invariant, complete, initialized) system is minimal if 
and only if it is canonical. 

Proof. Assume that £ is minimal. If it is not observable, then its observable 
reduction (cf. Lemma 6.8.3) realizes the same behavior and has fewer states, a 
contradiction. If it is not reachable from its initial state a; , then the restriction 
to TZ(x°) gives a system with the same behavior and fewer states. 

Conversely, assume that £ is canonical. If it is not minimal, then there 
exists another realization £' of lower cardinality. Reducing if necessary by 
observability or reachability as in the above paragraph, we may assume that 
£' is canonical. By the isomorphism Theorem given above, £ and £' must be 
isomorphic, contradicting the cardinality assertion. I 

Note that the proof also shows that, if there exists any realization with finite 
cardinality, then there is a canonical one of finite cardinality. 
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Example 6.8.6 We consider again the parity check example discussed in Ex- 
ample 2.3.3. In particular, we shall see how to prove, using the above results, 
the last two claims in Exercise 2.3.4. The behavior to be realized is A(r, 0, u>) = 

1 if lu(t — 3) + lu(t — 2) + uj(t — 1) is odd and 3 divides r > 
otherwise 

and we take the system with 

X := {0,1,2} x {0,1} 

and transitions 

V((i,j),l) := (i+ 1 mod 3,j + l mod 2) 

for i = 1 , 2 and 

V((0,j),l) := (1,0- 

The initial state is taken to be (0,0), and the output map has h(i,j) = 1 if 
i = and j = 1 and zero otherwise. (The interpretation is that (k, 0) stands for 
the state "t is of the form 3s + k and the sum until now is even," while states 
of the type (fc, 1) correspond to odd sums.) This is clearly a realization, with 6 
states. To prove that there is no possible (time-invariant, complete) realization 
with less states, it is sufficient to show that it is reachable and observable. 

Reachability follows from the fact that any state of the form (0, j) can be 
obtained with an input sequence jOO, while states of the type (1, j) are reached 
from x° using input j (of length one) and states (2,j) using input jO. 

Observability can be shown through consideration of the following controls 
LOij, for each (i,j): 

Woi := o, ^oo := 100, uj w := 10, wn := 00, w 2 i := 0, uj 20 := . 

Then, woi separates (0, 1) from every other state, while for all other pairs (i, j) ^ 
(0,1), 

\i,j)(Va0) = 1 

if and only if (i,j) = (a, (3). □ 

Exercise 6.8.7 Let A be any time-invariant complete behavior and let ~ be 
the following equivalence relation on CI: 

u ~ u> <£=> \((x>v) = X(u> v) W 

(to be more precise, one should write the translated version of v). This is the 
Nerode equivalence relation. Prove that A admits a finite-cardinality realization 
if and only if there are only finitely many equivalence classes under the Nerode 
relation. {Hint: It only takes a couple of lines, using previous results.) □ 
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The result in the Exercise can be interpreted in terms of generalized Hankel 
matrices. Consider the matrix with rows and columns indexed by elements of 
f2, and with \{uiv) in position (u>, v). Then finite realizability is equivalent to 
this matrix having only finitely many rows, and the number of different rows is 
equal to the cardinality of the state space of a minimal realization. This is in 
complete analogy to the linear case. 



6.9 Notes and Comments 

Basic Observability Notions 

The concept of observability originates both from classical automata theory 
and from basic linear systems. The material on final-state observability can 
be generalized to certain types of infinite systems, which in the continuous- 
time case give, because of Proposition 6.1.9, results about observability (see, for 
instance, [358] and [384]). 

The text [52] has an extensive discussion of the topic of multiple experiments, 
final-state determination, and control for finite systems. Also, [97] addresses 
the observability question for such systems. In the context of VLSI design a 
central problem is that of testing circuits for defects, and notions of observability 
appear; see, for instance, [148] and the references therein. 

Observability of Time-Invariant Systems 

The concept of observability, its duality with controllability for linear systems, 
and the notion of canonical realization, all arose during the early 1960s. Some 
early references are [155], [215], and [320]. 

In fact, Kalman's original filtering paper [214] explicitly mentions the du- 
ality between the optimization versions of feedback control (linear-quadratic 
problem) and observer construction (Kalman filtering). The term "canonical" 
was used already in [217], where one can find reachability and observability de- 
compositions. For related results on duality of time- varying linear systems, see 
[435]. 

For more on the number of experiments needed for observability, and in par- 
ticular Exercise 6.2.3, see, for instance, [97], [125], and especially [156]. See also 
[6] for continuous-time questions related to this problem. Results on observabil- 
ity of recurrent "neural" nets (the systems studied in Section 3.8, with linear 
output y = Cx) can be found in [14] . 

In the form of Exercise 6.2.12 (which, because of the simplifying assumption 
that the numbers u>i are distinct, could also have been easily proved directly), 
one refers to the eigenvalue criterion for sampling as Shannon's Theorem or the 
Sampling Theorem. It gives a sufficient condition to allow reconstruction of the 
signal r\ and is one of the cornerstones of digital signal processing. The result 
can be generalized, using Fourier transform techniques, to more general signals 
rj, containing an infinite number of frequency components. 
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It is possible to develop a large amount of the foundations of time-invariant 
systems based on the notion of observables associated to a system; see [373]. 

Linearization Principle for Observability 

Often one refines Definition 6.4.1 to require that states be distinguishable with- 
out large excursions. One such possibility is to ask that for each neighborhood 
W of x° there be a neighborhood V so that every state in V is distinguishable 
from x° using a control that makes the resulting trajectory stay in W. This 
is more natural in the context of Lyapunov stability, and can be characterized 
elegantly for continuous-time smooth systems; see, for instance, [185] and [365]. 
Other nonlinear observability references are, for instance, [5], [7], [28], [139], and 
[310]. 

Realization Theory for Linear Systems 

There are papers on realization of time- varying linear systems that use tech- 
niques close to those used for the time-invariant case. See, for instance, [235] 
and [434]. 

Recursion and Partial Realization 

See, for instance, [180], [181], [192], [395], and the many references therein, as 
well as the early paper [225], for further results on families of systems. 

Small perturbations of the Markov parameters will result in a nonrealizable 
sequence, since all of the determinants of the submatrices of TL are generically 
nonzero. In this context it is of interest to look for partial realizations, in which 
only a finite part of the sequence is matched; this problem is closely related to 
classical mathematical problems of Pade approximation. Lemma 6.6.9 is one 
result along these lines; see, for instance, [19] and the references therein, for 
a detailed treatment of partial realization questions and relations to problems 
of rational interpolation. The problem can also be posed as one of optimal 
approximation of Hankel operators; see the by now classical paper [3]. In ad- 
dition, the procedures that we described are numerically unstable, but various 
modifications render them stable; a reference in that regard is [110]. A recursive 
realization procedure is given in [18], which permits realizations for additional 
data to make use of previous realizations. 

Since a Markov sequence is specified in terms of an infinite amount of data, 
one cannot expect to solve completely the question of realizability unless some 
sort of finite description is first imposed. It is known however that, for arbitrary 
"computable" descriptions, the problem of deciding realizability is undecidable 
in the sense of logic, that is, there is no possible computer program that will 
always correctly determine, given a description of a Markov sequence, whether 
a realization exists; see, for instance, [356]. 
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Rationality and Realizability 

The ideas of realization and relations to input/output equations go back at 
least to the nineteenth century, in the context of using integrators to solve 
algebraic differential equations; see [400] as well as the extensive discussion in 
[212], Chapter 2. The relations between rationality and finite Hankel rank are 
also classical (for the scalar case) and go back to the work of Kronecker (see 
[152]). 

Algebraic techniques for studying realizations of linear systems were empha- 
sized by Kalman; see, for instance, [220], [221], and [223], as well as [228] for 
relations to econometrics. 

Proposition 6.7.6 could also be proved using Laplace transform techniques; 
we used a direct technique that in principle can be generalized to certain nonlin- 
ear systems. For these generalizations, see, for instance, [360] and [421], which 
establish that certain types of i/o behaviors satisfy polynomial equations 

E{y{t+n),y{t+n- 1), . . . ,y(t),u(t + n - 1),... ,«(*)) = (6.44) 

(or E{y i - n \t), J/ (n_1) (t), ■■■, y(t),u( n -V(t), . . . , u{tj) = 0) if and only if they are 
realizable, for discrete-time and continuous-time systems, respectively. In the 
nonlinear case, however, not every "causal" i/o equation necessarily gives rise 
to an i/o behavior, and this in turn motivates a large amount of research on 
such equations; see, for instance, [408]. Some authors, motivated by differential- 
algebraic techniques, have suggested that realizability should be defined in terms 
of i/o equations; see especially [141] and the references therein. 

The reference [272] discusses i/o equations for nonlinear systems in the con- 
text of identification problems. Related material is also presented in [307], which 
uses i/o equations (6.44) in which E is not linear, nor polynomial, but instead 
is given by iterated compositions of a fixed scalar nonlinear function with lin- 
ear maps. Numerical experience seems to suggest that such combinations are 
particularly easy to estimate using gradient descent techniques, and they are in 
principle implementable in parallel processors. The name "neural network" is 
used for this type of function E, because of the analogy with neural systems: 
the linear combinations correspond to dendritic integrations of signals, and the 
scalar nonlinear function corresponds to the "firing" response of each neuron, 
depending on the weighted input to it. 

Realization and i/o equations can also be studied in a stochastic context, 
when u(0),u(l), . . . in (6.43) are random variables. In that case, especially 
if these variables are independent and identically distributed, (6.43) describes 
what is called an ARMA or aide/regressive moving average model for the stochas- 
tic process, in the stochastic systems and statistics literature. (The outputs 
form a time series, and the realization is a Markov model for the series.) See, 
for instance, the "weak" and "strong" Gaussian stochastic realization problems 
studied, respectively, in [131], [132], [222], [409], and [10], [11], [280], and the 
related [344]. 
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Abstract Realization Theory 

It is possible to develop a unified theory of realization for discrete-time finite 
systems and linear finite dimensional systems, in the language of category the- 
ory; see, for instance, [22]. There are also many other approaches to realization 
theory, for instance for nonlinear systems evolving on manifolds in continuous- 
time ([199], [383], and [389]) or in discrete-time ([203]), systems on finite groups 
([75]), polynomial discrete-time ([360]) and continuous-time ([38]) systems, bi- 
linear systems ([66], [198], [359], and [382]), or infinite-dimensional linear sys- 
tems ([316], [437], and [438]). The goal in these cases is to study realizations 
having particular structures — such as linear realizations when dealing with lin- 
ear behaviors — and to relate minimality (in appropriate senses) to various no- 
tions of controllability and observability. It is also possible to base a realization 
theory on generalizations of Hankel matrix ideas to bilinear and other systems; 
see, for instance, [137] and [138], as well as applications of these ideas in [96]. 
One also may impose physical constraints such as in the study of Hamiltonian 
control systems in [106] and [204]. 

In principle, the complete i/o behavior is needed in order to obtain a real- 
ization. However, if the system is "mixing" enough, so that trajectories tend 
to visit the entire state space, one may expect that a single long-enough record 
will be sufficient in order to charaterize the complete behavior. In this sense one 
can view the recent work identifying dynamics of chaotic systems from obser- 
vation data (see, e.g., [85] and [130]) as closely related to realization theory (for 
systems with no controls), but no technical connections between the two areas 
have been developed yet. 
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Chapter 7 

Observers and Dynamic 
Feedback 



Section 1.6 in Chapter 1 discussed the advantages of using integration in order 
to average out noise when obtaining state estimates. This leads us to the topic 
of dynamic observers. In this chapter, we deal with observers and with the 
design of controllers for linear systems using only output measurements. 

7.1 Observers and Detectability 

In this section we first discuss briefly the construction of observers for continu- 
ous-time linear systems and then study the more basic questions raised by this 
procedure. The next subsection can be skipped if desired, as all results are 
proved later, but it provides the intuitive ideas underlying everything else. 

Motivational Discussion 

Consider the time-invariant continuous-time system 

x = Ax + Bu 
y = Cx 

and the problem of reconstructing the state x(t) from the measurements y(t). 
One solution to this problem when (A, C) is observable is to apply the optimal 
integral operator M used in Theorem 25 (p. 279). For each fixed interval, let us 
say of the form [0, T], this provides the state x(0) that produced a given output 
function y(-) on that interval. From the knowledge of x(0) and the control u(-), 
one may then recover the entire state trajectory. 

In a sense, this is an "open-loop" procedure, and it has the same disad- 
vantages that open-loop control has. For instance, if there is a possibility that 
the state could change due to unmeasured disturbances acting on the system, 

315 
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it is necessary to repeat the estimation procedure. Just as in the discussion 
about the advantages of feedback control, one can argue that a recursive esti- 
mation procedure — in which the estimates are updated continuously, as more 
observations are received — is often more useful. Again, as in that case, the 
advantages of recursive estimation could be quantified if a model incorporating 
disturbances is used. In fact, in analogy to the solution to the linear-quadratic 
problem (Chapter 8), it turns out that this type of estimation is indeed opti- 
mal with respect to certain optimization criteria. But again, as with feedback 
control, we take here the direct route of simply posing recursive estimation as 
a problem to be studied in itself. 

One obvious way to obtain an estimate for the state of the above linear 
system is to start with a model 

z(t) = Az(t) + Bu(t) 

of the system, initialized at any state, to which the same controls are applied. 
If we are lucky enough that a;(0) = z(0), then it will follow that the solutions 
coincide for all t, and a perfect estimate will be maintained. But of course the 
problem is that x(0) is unknown; the only information available about x is the 
measurement y. We argue then as follows. The quantity 

d(t) := Cz(t) - y(t) 

measures the discrepancy between the actual measured value and the output 
value that would result if our estimate z were indeed the correct state. This 
error can be computed from the available data, so it can be used to influence the 
dynamics of z. To keep matters simple, we restrict attention to lin ear functions 
of the error. That is, we propose an estimator 

z(t) = Az(t) + Bu(t) + Ld(t) , 

where L is a p x n matrix still to be decided upon. Let e(t) := z(t) — x(t) be 
the difference between the actual and estimated state trajectories. Immediately 
from the equations for x and z we conclude that the differential equation 

e(t) = (A + LC)e(t) 

holds for e. If it is possible to choose a matrix L so that A + LC is Hurwitz, 
then e converges to zero for all initial conditions. In terms of x and z, this 
means that, whatever the initial state and the initial estimate, the estimation 
error decreases asymptotically, in fact at an exponential rate. Since the matrix 
A + LC is Hurwitz if and only if A' + C L' is, such an L can be found whenever 
the pair (A',C) is asymptotically controllable, and in particular if (A, C) is 
observable. The rest of this section will only expand on this simple construction, 
showing that if the pair (A' ', C") is not asymptotically controllable then nonlinear 
estimators cannot work either, and will pose the definitions in a somewhat more 
abstract context, applying to general systems. Relatively little is known as yet, 
however, about estimators for nonlinear systems. 
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Asymptotic Observability 

Let S be a time-invariant complete system with outputs and assume that E 
is topological (Definition 3.7.1), that is, X is a metric space and paths depend 
continuously on initial states. Assume that u° € XI, y° € V, and the state 
x°, are such that (x , u ) is an equilibrium pair and h(x ) = y°. (Recall that 
by equilibrium pair we mean that for every t G T + , (f>(t,0,x Q ,u>°) = x° when 



X 



:= {xe1\ \°/{uj ) = y° for all t € T+} , 



the set of states indistinguishable from x° when using identically u° controls. 
This set is invariant under such controls = u , so there is a well defined system 
E° (which depends on the particular u°,y° that was picked) with state space 
X°, control-value space {u }, and whose transition map is the restriction of <j>. 
This is a classical dynamical system (no inputs nor outputs). 

Definition 7.1.1 The time-invariant complete system E is asymptotically 
observable (with respect to x°,u°,y ) if E is globally asymptotically stable 
(with respect to x a ). D 

In informal terms, this is the property that, if a state gives rise to an output 
constantly equal to y° when the constant control u° is applied, then the resulting 
trajectory must converge to x° (and if the initial state started near x , it never 
goes very far). The more standard terminology (at least for linear systems) is 
to say that the system is detectable. 

Let E be a linear (finite dimensional) system over I = 1 or C. We say 
simply that E is asymptotically observable (or equivalently, detectable) if it is 
so with respect to 

x° = u° = y" = . 

In this case E° is the linear system 

x = Ax 

or in discrete-time x + = Ax, evolving on the state space O, where O is the set 
of states indistinguishable from 0, the same as the kernel of 0(A,C). Thus, 
detectability means precisely that 

cm = o =► m -» o 

for all solutions of x (or x + ) = Ax. In terms of the observability decomposition 
in Equation (6.8), C£ = implies that the first r components of such a solution 
£ must be identically zero (since the pair (A\,Ci) is observable), and therefore 
detectability is simply the requirement that A 3 be a Hurwitz (or convergent) 
matrix. We conclude: 
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Lemma 7.1.2 For linear time-invariant continuous-time or discrete-time sys- 
tems, (A, C) is asymptotically observable if and only if (A' , C) is asymptotically 
controllable. □ 

Note that the Hautus condition for asymptotic controllability becomes that 
(A, C) is asymptotically observable if and only if 



rank 



XI - A 
C 



n VAeC,ReA>0 



for continuous-time systems, and similarly (with the condition now being for 
|A| > 1) in discrete-time. 

Observers 

Still, £ denotes an arbitrary complete time- invariant system with outputs. An 
observer £ for £ (more precisely, this should be called a "final-state asymptotic 
observer" ) is a system that produces an estimate of the current state of £ based 
on past observations. At any instant, the inputs to £ are the current inputs 
and outputs of £, and the estimate is obtained as a function of its state and the 
current observation. 

Definition 7.1.3 A (strong) observer for £ consists of a (time-invariant and 
complete) system £ having input value space 

together with a map 

6: 2x^1 

(where Z is the state space of £j so that the following property holds. For each 
iGl, each z € Z, and each to £ 111 ' 00 ), we let £ be the infinite path resulting 
from initial state x and control u>, that is, 

£(t) = (f>(t,0,x,u)\[o it )) 

for allt G T +7 and write rj(t) = h(£(t)); we also let £ be the infinite path resulting 
from initial state z and control (w,r?) for the system £, 

C(t) = ${t,0, z,{w\[ ,t),v\[o,t))) , 
and finally we write 

i(t):=e(((t)Mt))- 

Then it is required that, for every x, z and every to, 

d U{t),i{t)) ^0 as t^oo (7.1) 
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(global convergence of estimate) as well as that for each e > there be some 
5 > so that for every x, z, lu 

d (x, 0(z, h(x))) < 8 => d U(t),i(tj) <£foralU>0 (7.2) 

(small initial estimate results in small future errors). □ 



£,e 



Figure 7.1: Observer. 

By a linear observer for a linear (time-invariant, finite dimensional) system 
E we mean a linear system E together with a linear map 0. 

Theorem 31 Assume that E is a linear system (finite dimensional, time-in- 
variant, over R or C). Then the following properties are equivalent: 

1. There is an observer for E. 

2. There is a linear observer for E. 

3. E is asymptotically observable (i.e., detectable). 

Proof. Note that Property 2 implies Property 1. We prove that 3 implies 2. 
For this, it is only necessary to take E of the form 

z[orz + ] = (A + LC)z + Bu-Ly, (7.3) 

as discussed earlier, and 0(z,y) := z. By Proposition 5.5.6 and Lemma 7.1.2, 
there is always a choice of L that insures that A + LC is a Hurwitz or convergent 
matrix, respectively. Since for every initial x, z and control lu one has that 
e(t) := z(t) — x(t) satisfies 

e[ore+] = {A+LC)e, 

an observer results. 

Conversely, assume that Property 1 holds. Pick an observer and let z be any 
initial state. We claim first that, if 

C(t) = 4>(t,o,z,(o,o)), 

then #(£(£), 0) — > 0. This is because of Property (7.1) applied to the trajectory 
lu = 0, £ = of E. Now let £ be any trajectory corresponding to the control 
lu = for which C£(t) = 0, as in the definition of detectability. Since £ is the 
same as when £ = 0, it follows that again #(£(£), 0) — > 0, so (7.1) now implies 
that £(i) -» 0. I 
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Remark 7.1.4 When the system E is discrete-time and observable, one may 
pick a matrix L so that A + LC is nilpotent, by the Pole-Shifting Theorem. 
Thus, in that case a deadbeat observer results: The estimate becomes exactly 
equal to the state after n steps. □ 

Note that the above construction results in an observer of dimension equal 
to that of the original system. In general, one may also construct reduced order 
observers, in the following sense. Assume for simplicity that C has rank p. Then, 
— after, if necessary, changing variables in the state space and the output- value 
space — one may assume that 

C = (I 0) . (7.4) 

This means that p components of x can be measured instantaneously, which 
leads one to expect that an observer of dimension n — p will exist. The following 
exercise establishes that fact. 

Exercise 7.1.5 Let E be a detectable continuous-time time-invariant system, 
with C of the form in Equation (7.4). Show that there exists a linear observer 
of dimension n — p. (Hint: Argue as follows. First write the equations of E in 
block form 

±i = Anxi + A 12 x 2 + Biu 
x 2 = Ai\X\ + A22X2 + B 2 u , 

so that y = x\ and note that the pair (A22, ^12) is again detectable — this takes 
one line to prove. Next think of the second equation as defining a system whose 
input is A21X1 + B 2 u, and consider an observer 

q = (A22 + LAi 2 )q - LA12X2 + A21X1 + B 2 u 

for it. The reason that this does not yet provide a solution is that it involves 
the term LA12X2, which is not a measured quantity. However, an equation for 
z := q + Lxi can be derived, that does not contain any such term. Fill in the 
details and write explicitly the form of the observer (E, 0).) □ 

The form of the observer in the proof of Theorem 31 is the most general 
possible if one insists that 8(z,y) = z: 

Exercise 7.1.6 Assume that E = (A,B,C) is a continuous-time linear system 
admitting a linear observer 

z = A z + B u + Ny 

with 8(z,y) = z. Show that it must be the case that Bq = B, Aq = A — NC, 
and that Aq must be a Hurwitz matrix. □ 
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Remark 7.1.7 Consider the observer constructed in the proof of Theorem 31, 
say in continuous-time. The rate of convergence to zero of the error e(t) will 
be controlled by how negative the real parts of the eigenvalues of A + LC are. 
In order for this convergence to be fast, L must be large in magnitude. If the 
observation y has been corrupted by noise, say y = Cx + d, where d cannot be 
measured by the observer, then the error equation becomes 

e(t) = (A + LC)e(t)-Ld(t), 

which tends to remain large if d is persistent and L is large. What this means 
is that there is a trade-off in achieving small error: large L may be needed for 
fast convergence if there is no noise, but small L amplifies less the effect of 
noise. This trade-off can be formulated precisely as an optimization problem. 
A Kalman filter is an observer that has been optimized with respect to uncer- 
tainty in observations as well as, if desired, state disturbances. Mathematically, 
the problem of designing Kalman filters turns out to be reducible, by duality, 
to a linear quadratic optimal control problem, and has been one of the most 
successfully applied parts of control theory. A deterministic version is provided 
in Section 8.3. □ 
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We turn now to the question of stabilizing systems based only on output mea- 
surements. As the discussion in Section 1.2 regarding proportional-only control 
illustrated, direct output feedback alone in general is not sufficient. 

Example 7.2.1 As an example of the insufficiency of output feedback, take 
the system 

±i = x 2 
x 2 = u 

with output map y = x\. Even though this system is controllable and observ- 
able, we claim that there exists no possible continuous function 

k:R^R 

that depends only on y and stabilizes in the sense that, for each x = (x\,X2) 
and every solution £ = (^1,^2) of 

±1 = x 2 
x 2 = fc(xi) 

with £(0) = x, the solution is defined for all £ > and £(£) — > 0. If any such k 
existed, we could consider the smooth ("energy") function 

V(x) := x\ - 2 / k(a)da, 
Jo 
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which is constant along all trajectories. Since all trajectories converge to the 
origin by assumption, it follows that V must be constant and hence equal ev- 
erywhere to V(0) = 0, contradicting V(0, 1) = 1. □ 

Exercise 7.2.2 Show that, if A has zero trace and if CB = 0, then A + BFC 
(the matrix obtained by feeding u = Fy to x = Ax + Bu, y = Cx) cannot be 
Hurwitz for any F. □ 

This section deals with the use of dynamic feedback, that is, the control of 
systems using other systems (as opposed to static feedback laws). As discussed 
in Chapter 1, this can be interpreted, in the continuous-time case when con- 
tinuous-time systems are used as controllers, as the use of feedback laws that 
involve integrals of the observations. In order to avoid technical problems that 
arise only for nonlinear systems, and because the results to be presented apply 
mainly in the linear case, we again restrict attention to complete and time- in- 
variant systems S. 

We start by defining the interconnection of two systems with output Si and 
S 2 . It is understood that both time sets must be the same. Subscripts i are 
used to indicate objects associated with system Ej. 

Definition 7.2.3 Let Y,\ and E2 be two systems, and let 

h : Vi xy 2 ->Ui, » = 1,2 

be two maps. We say that the interconnection of Si and £2 through k\ 

and hi is well-posed if for each (xi, X2) € Xi X X2 there exist unique functions 

& : [0, 00) -> Xi , i = 1,2 
such that, with the notations 

m{t) ■= hi(ti(t)) 

u>i(t) := ki(r)i(i),r)2(i)) 
it holds that £j = ipi(xi,LUi) for i = 1,2. □ 

Recall that £j = ipi(xi,u>i) means that £j(£) = 4>i(t,0,Xi,tOi\[Q t )) for all t G 

Lemma/Exercise 7.2.4 If the above interconnection is well-posed, then defin- 
ing X := Xi x X 2 and 

<Kt,0,(x U X2)) = (tl(t),&(t)) 

results in a complete time-invariant system E with no controls. □ 
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Figure 7.2: Interconnection of two systems. 

We call the system in Lemma 7.2.4 the interconnection of Si and S 2 with 
connecting maps ki,k 2 - When Si and S2 are topological, this interconnection 
is a topological system (with the product metric on Xi x X2) if the functions £j 
in Definition 7.2.3 depend continuously on (xi,cC2) uniformly on compact time 
intervals. In this case, we say that the interconnection is topologically well-posed. 

For continuous-time systems Si and S 2 and any maps k\,k 2 , well-posedness 
means that solutions of 

X\ = fi(xi,k 1 (hi(x 1 ),h 2 (x 2 ))) 
X2 = f2{x2,k 2 {h 1 {xi),h 2 {x 2 ))) 

must exist and be unique for all initial states and all t. If the systems as well 
as k\ and k 2 are of class C , then uniqueness always holds, and the only issue is 
completeness. Topological well-posedness is also automatic in that case. 

When the system S2 has just one state, we can ignore the argument r\ 2 in 
k\, since it is constant, as well as the function £2 (see Figure 7.3). Then a well- 
posed interconnection is nothing more than an output feedback law u = k(y) 
for which the closed-loop system is complete. 



Figure 7.3: Static feedback case of interconnection. 



If Si and S2 are discrete-time systems, the interconnection is always well 
defined, as the corresponding difference equations always have a solution. 

Definition 7.2.5 Let Si be a topological system, and pick any x\ G X. The 
system Si is dynamically stabilizable (with respect to x\) by (S2, x 2 , fci, k2), 
where S2 is a topological system, if the interconnection of Si and S2 through 
k\, fc 2 is topologically well-posed and is globally asymptotically stable with respect 
to{x\,x%). □ 
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For instance, any controllable linear system with C = I can be dynamically 
stabilized with a linear £2 of dimension and k\ linear. The next result shows 
that, when C ^ I but the system is observable, one still can obtain stabiliz- 
ability. The idea is very simple: If u = Fx stabilizes the system, then we feed 
u = Fz, where z is an estimate of x calculated by an observer. 

By a linear stabilizer (£2, ^2; ^i> ^2) for a (continuous-time or discrete-time) 
system Si we mean that £2 is a (continuous-time or discrete-time, respectively) 
system, that both maps ki are linear, and that x® = 0. 

Theorem 32 Assume that £1 is a linear system (finite dimensional, time-in- 
variant, over K or C). Then the following properties are equivalent: 

1. There is a dynamic stabilizer for £1 (with respect to x° = 0). 

2. There is a dynamic linear stabilizer for £1 (with respect to x a = 0). 

3. £1 is asymptotically controllable and asymptotically observable. 

Proof. We first prove that Property 1 implies Property 3 for any system (not 
necessarily linear). More precisely, assume that the interconnection of £1 and £2 
through fci, &2 is topologically well-posed and asystable with respect to (x\, x®), 
and let 

y?:=hi(x?) and u? := fci(y?,yg) i = l,2. 

Since (a^, x%) is an equilibrium point for the interconnection, it follows that 
(x?, u?) is an equilibrium pair for the system £,. Then the claim is that £1 must 
be asymptotically controllable to x\ and detectable with respect to x\,u\,y®. 
(An analogous statement holds for £2, of course.) Asymptotic controllability is 
immediate from the definitions, so we prove the detectability statement. 

Consider any state iG Xi indistinguishable from x\ when using identically 
u\ controls, and let £1 := ip(x,w1) be the resulting infinite time path. Let 
£2 = xjj. We claim that (£1,^2) must be a trajectory of the interconnection. 
This is true by the uniqueness statement in the definition of well-posedness; 
observe that 771 = y® because of the indistinguishability assumption on x and 
that r\i = y,2- Thus, (£1,^2) converges to (x^a;^), which implies that also £1 
converges to x\, as wanted. Moreover, if x starts near x\, then (£1,^2) remains 
uniformly near (x^xjj), again by asymptotic stability of the interconnection, so 
£1 remains near x\. 

We have proved that the system £ is detectable with respect to x\,u\,yi. 
When, as in the statement of the Theorem, £1 is a linear system and x° = 0, 
and using that y® = fci(0) = 0, we have then that the following property holds 
(using for definiteness the continuous-time case, but the discrete-time argument 
is exactly the same): for every state trajectory £(£) so that £ = ^4£ + Bu\ and 
C£(t) = 0, necessarily £(£) — > as t — > 00. On the other hand, (0,Ui) is an 
equilibrium pair, so = A0 + Bu\, which implies that Bu\ = 0. Thus it is 
also true that for every state trajectory £(£) so that £ = At; and C£(t) = 0, 
necessarily £(i) — > as t — > 00, and this means that the system is detectable 
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with respect to 0,0,0, which is what we mean when we say that the linear 
system Si is asymptotically controllable. 

It is only left to show that, for linear systems, Property 2 follows from 
Property 3. Let (A,B,C) be the triple associated to Si and pick matrices 
F, L such that A + BF and A + LC are both Hurwitz (in continuous-time) or 
convergent (in discrete-time). Pick £2(2/1, 2/2) = 2/1 an d &i(?/i;2/2) = J/2, and the 
system £2 defined by 

{A + LC + BF, -L, F) 

so that the resulting interconnection is the linear system with no controls and 
"A" matrix equal to 

A BF \ X (A + BF BF \ 

-LC A+LC+BFj \ A+LCJ 

where 

- - {!, ! 

It follows that the interconnection is asymptotically stable. 
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Figure 7.4: Observer/controller configuration in proof of Theorem 32. 

Exercise 7.2.6 Let £1 be a continuous-time system of class C , 

x = f{x,u), y = h(x), 

and assume that (0,0) is an equilibrium pair and /i(0) = 0. Let S* be the 
linearization about the zero trajectory, and assume that S* is asymptotically 
controllable and asymptotically observable. Show that then there exist matrices 
F, D, E (of suitable sizes) such that the origin of 

x = f{x,Fz) 
z = Dz + Eh(x) 

is locally asymptotically stable. This corresponds to a notion of local dynamic 
stabilizability □ 

Exercise 7.2.7 Refer to Exercise 3.2.12. Take all constants equal to one, for 
simplicity. 
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(a) Using y = h as the output, find a dynamic feedback stabilizer for E. 

(b) Show that there is no static output feedback u = Fy which stabilizes the 
system. 

(c) (Optional) Show that it is possible to find a stabilizing feedback law for 
this example which depends only on h and <j), i.e., there are coefficients 
/!, v so that with u = [ih + v<f> the closed-loop system is stable. (This is 
of interest in practice, because h and <f) can easily be measured with an 
altimeter and gyro, respectively.) □ 

Exercise 7.2.8 Refer to Exercise 3.2.13, and take for simplicity M = m = F = 
9 = 1 = 1- 

(a) Using y = xi = 5 as the output, find a dynamic feedback stabilizer for E. 

(b) Using both xi and x% as measured (see Exercise 6.2.6, part (c)) and as- 
suming instead m = 0, find a dynamic feedback stabilizer. 

(c) (Optional) Show that for no possible "proportional feedback" control law 
u = fx = ax i + /3x3 is the closed- loop system stable. (You may want to 
use the fact, easily proved, that a monic polynomial can be Hurwitz only 
if all its coefficients are strictly positive.) □ 

7.3 External Stability for Linear Systems 

There are many nonequivalent definitions possible for the concept of stability of 
i/o behaviors. Rather than dealing with these in a general setting, however, we 
study only the linear (finite dimensional) case, in which all definitions coincide. 
Let A be either a linear time-invariant discrete-time behavior or a linear 
time-invariant continuous-time integral behavior (Definitions 2.4.3 and 2.7.8, 
respectively), with K = 1 or C. Let K be the impulse response of A. In the 
discrete-time case, this is a Markov sequence; in continuous-time it is a matrix 
of functions K € £?^ c (0, oo). For each lo € VS ' 00 ' (locally essentially bounded 
in the continuous-time case), we can consider the infinite time output rj € ty°'°°> 
produced by lo, 

r/(t) := X(t,0M[0,t)) ■ 

Consistently with the discussion in Remark 2.2.2, we will denote this function 
r\ as A(oj). Formulas (2.17) and (2.27) say that 

X(lu) = K * lo , 

where * denotes convolution in discrete-time or continuous-time. Note that in 
continuous-time, X(uj) G £^°' (oc (0, oo). 

For notational simplicity when giving unified statements for continuous-time 
and discrete-time, for each integer r we let C^° denote not just the set of essen- 
tially bounded measurable functions [0, oo) — > K r (K = R or C, depending on 
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whether we are looking at real or complex systems), but also the set (7°°) r of 
all sequences 

v = yo,yi,V2,V2,--- 

of vectors in W satisfying 

IMIoo := su PlMI < °°- 

In general, X(u>) is not necessarily in £2° even if lu G jC™. When it is, we may 
view A as an operator 

A ■ L 'm ^-p ■ 

This operator is continuous with respect to the essential supremum norms if 
and only if it is bounded, that is, if there is some constant 7 so that 

HamL < 7 IMIoo 

for all lu. In that case, one says that the behavior A is uniformly bounded- 
input bounded-output, or UBIBO for short. 

For an impulse response K, we let ||^T|| 1 denote its (possibly infinite) C 1 
norm. By this we mean 



\K(t)\\ dt 



in continuous-time, or 



£11* 



in discrete-time, where ||-|| denotes the operator norm in K pxm induced by 
Euclidean norm. We say that the impulse response is integrable if \\K\\ 1 < 00; 
this is equivalent to the requirement that each entry -fQj(-) of K be an integrable 
function. 

Lemma 7.3.1 The behavior A is UBIBO if and only if its impulse response K 
is integrable. 

Proof. Sufficiency follows from the fact that 

||AML < 11*11! IMIoo 

for all d. To establish necessity, take any fixed i,j; we will prove that Kij is 
integrable. Pick any T > and let lu be defined as follows. The coordinates 
different from j are identically zero, and the j'th coordinate is 

u>j(t) := sign Kij(T- t) 

for all t G [0, T] and zero otherwise (with the convention signO = 0). Then the 
ith coordinate of r\ = X(u>) satisfies (in continuous-time) 

m (T) = / \K l3 (t)\ dt 
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and similarly in discrete-time. Since the behavior is UBIBO, also 

\Vi(T)\ < IMIoo < 7l|w|loo < 7, 
which implies that 1 1 -K"» j 1 1 j < 7 < 00, as wanted. I 

One of the most useful facts about stable systems is that decaying inputs 
produce decaying outputs: 

Lemma 7.3.2 If A is UBIBO, then lim^oo u>(t) = implies that 

lim A(w)O) = 0. 

t — >oo 

Proof. Assume that u) — > 0, let r\ := A(w), and pick any e > 0. We wish 
to find a T such that ||t?(£)|| < e for all t > T. The calculations will be done 
for continuous-time, but the discrete-time case is entirely analogous. Denoting 
a := llwllgo and 7 := \\K 1^, there is some To so that 

l|w(*)|| < f 
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for all t > Tn and 



\\K(t)\\ dt < ±- 
T 2a 



We pick T := 


2T ; 


with this choice, for each t > T it holds that 


\\v(t)\\ 


< 


aj^ °\\K(t-r)\\dT +^J t \\K(t-r)\\dr 




= 


a f \\K(t)\\ dr + f f ° ||if(r)|| dr < e 

Jt-T 2 7 J 


as desired. 







From now on, let E be any (finite dimensional, time-invariant) discrete-time 
or continuous-time system E = (A, B, C) over K = R or C. We shall say that E 
is internally stable if A is a convergent or Hurwitz matrix, respectively. Given 
any such system, we let Ag be its i/o behavior, and Ws the generating series 
associated to its Markov sequence. 

Proposition 7.3.3 If E is internally stable, then As is UBIBO. 

Proof. In continuous-time, each entry of the matrix K(t) = Ce* B is a sum of 
terms of the form t l e xt , with all A having negative real parts, and this implies 
that K is integrable. In discrete-time, Ki = CA t ~ 1 B, and terms of the form 
t l X l appear, |A| < 1, so the same is true. I 

The following characterization of asymptotic observability will be useful be- 
low. 
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Lemma 7.3.4 The system E is detectable if and only if the following property 
holds: For each solution £ of X = Ax (or x + = Ax), 

cm — * o =► m — * o. 

£ — »oo £ — »oo 

Proof. The property clearly implies detectability (C£(£) = => £(£) — * 0). 
Conversely, assume that E is detectable and let 

z[ovx + ] = {A+LC)z+Bu- Ly (7.5) 

be an observer constructed as in the proof of Theorem 31 (p. 319), where A+LC 
is a Hurwitz or convergent matrix, respectively. Assume that C£(£) — > 0. We 
consider the system (7.5) with output z, and apply, to this system, Proposition 
7.3.3 and Lemma 7.3.2. We conclude that the solution £ of (7.5) with 

y (t) = 0^) , u = , and C(0) = 

must satisfy £(£) — > 0. Since, by definition of observer, £(£) — £(<) — » 0, the 
conclusion follows. I 

A converse of Proposition 7.3.3 is provided by the following result: 

Theorem 33 If E is canonical and As is UBIBO, then E is internally stable. 

Proof. We provide details only for continuous-time, as the discrete-time case 
is similar. Pick any solution £ of X = Ax, and let a; := £(1). We need to see 
that at) ~ * 0- By controllability of E, there is some control lu on [0, 1] steering 
to x°. We extend lu by uo(t) = for t > 1; note that 

C£(t) = A(w)(t) 

for all t > 1. Since A s is UBIBO and uj{t) —> 0, necessarily C^(i) — > 0. By 
Lemma 7.3.4 it follows that ^ — > 0, using detectability of E. I 

Finally, one can characterize these properties in terms of rational represen- 
tations: 

Proposition 7.3.5 The behavior A^ is UBIBO if and only if there exists a 
rational representation Ws = q~ 1 P in which q is a Hurwitz (or convergent, in 
discrete-time) polynomial. 

Proof. Without loss of generality, we may assume that E is canonical (other- 
wise pick a canonical realization). If Ag is UBIBO, then E must be internally 
stable; since q may be chosen as the characteristic polynomial of A, it can be 
chosen to be Hurwitz or convergent, respectively. Conversely, if there is such 
a representation, then there is at least one realization that is internally stable. 
For instance, the observability form realization has a characteristic polynomial 
equal to q p , and hence this is an internally stable realization. I 
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Exercise 7.3.6 Prove that "E is asymptotically controllable and asymptoti- 
cally observable" is sufficient (instead of assuming that E is canonical) in The- 
orem 33. □ 

Exercise 7.3.7 Give an example of a linear system that is not internally stable 
but whose behavior is UBIBO. □ 

Exercise 7.3.8 Consider the following "non-uniform BIBO" property: For 
each bounded u>, A(cj) is also bounded (that is, A induces an operator C^ — > £2°, 
but this is not necessarily continuous). Show that for every E as above, As is 
UBIBO if and only if it satisfies this property. (Hint: Just read the above 
proofs.) □ 

Exercise 7.3.9 Refer to Exercise 3.2.12. Take all constants equal to one, for 
simplicity. Using y = h as an output, show that the i/o behavior of E is not 
UBIBO. Find an explicit input that produces a diverging output (when E starts 
at the zero state). □ 

Exercise 7.3.10 Repeat problem 7.3.9 for the example in Exercise 3.2.13, tak- 
ing M = m = F = g = l= 1. Use y = x\ = 6 as the output. □ 

Exercise 7.3.11 Denote by K, the class of functions [0, oo) — > [0, oo) which are 
zero at zero, strictly increasing, and continuous, by /Coo the set of unbounded 
K, functions, and by ICC the class of functions [0,oo) 2 — » [0,oo) which are of 
class K, on the first argument and decrease to zero on the second argument. A 
continuous-time time-invariant system x = f(x, u) with state space X = W l 
and control- value space IX = R m is said to be input-to- state stable (ISS) if there 
exist a function (3 G K,C and a function 7 € /Coo so that the following property 
holds: For each T > 0, if uj is a control on the interval [0, T] and £ is a solution 
of £ = /(£, u>) on the interval [0,T], then 

U(T)\\ < max {/?(||£(0)||,r), 7 (IMU}- (7-6) 

Show: 

1. Every ISS system is complete (i.e., every input is admissible for every 
state). (Hint: Use Proposition C.3.6.) 

2. For ISS systems, if u) is any input on [0, 00) which is bounded (respectively, 
converges to zero as t — > 00), and if £ solves £ = /(£, oj), then £ is bounded 
(respectively, converges to zero as t — > 00). 

3. If a system is ISS, then it is internally stable, in the sense that the system 
with no inputs x = f(x,0) is globally asymptotically stable (with respect 
to the origin). 
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4. A continuous-time time-invariant linear system is ISS if and only if it is 
internally stable. Moreover, for such systems one may always find f3 of 
the form /3(r, i) = c\e~ at r and 7 of the form 7(7") = c^r, for some positive 
constants C\, C2, and a. 

5. Show that internal stability does not imply ISS for nonlinear systems, by 
giving an example of a system for which the origin is globally asymptot- 
ically stable for x = f(x, 0) but for which there exists a bounded control 
lo and an unbounded solution of £ = /(£, 10). 

6. If a system is ISS, then it has finite nonlinear gain, meaning that there 
exists a function 7 £ K,^ so that the following property holds: If uj is any 
input on [0, 00) which is bounded, and if £ solves £ = f(^,uj), then 

3£(*)ll < 7(IMU- (") 

(The function 7 is then called a nonlinear gain.) □ 

Exercise 7.3.12 A continuous-time time-invariant system with outputs (but 
no inputs) x = f{x), y = h(x), having state space X = R™, and output- value 
space y = W, is said to be output-to-state stable (OSS) if there exist a function 
(3 £ K,C and a function 7 £ /Coo (terminology is as in Exercise 7.3.11) so that 
the following property holds: For each T > 0, if £ is a solution of £ = /(£) on 
the interval [0,T], and r){t) = h(£(t)) for t £ [0,T], then 

U(T)\\ < max{ / 3(||e(0)||,T), 7 (h|| co )}. (7.8) 

Show: 

1. A continuous-time time-invariant linear system is OSS if and only if it is 
detectable. 

2. In general, a system may be detectable (with respect to x° = and y = 0) 
yet not be OSS. □ 

7.4 Frequency-Domain Considerations 

The classical way to treat linear time-invariant finite dimensional systems is via 
frequency-domain techniques. In this section we briefly sketch some of the basic 
concepts of that approach, mainly in order to make the reader familiar with 
terminology and elementary facts. No attempt is made to cover this vast area 
in any detail. We restrict attention to continuous-time systems, though similar 
results can be derived for the discrete-time case. 

Let A be a realizable linear time-invariant behavior of integral type over 
K = R, and let K be its impulse response. Realizability means that there are 



332 7. Observers and Dynamic Feedback 

matrices A, B, C so that 

OO j 

Kit) = Ce tA B = YCA j B- 

3=0 J ' 

for all t > 0. Write Aj := CA j ~ 1 B, j = 1,2,... for the associated Markov 
sequence and W for its associated generating series. Pick any real number a 
larger than the norm of the matrix A; then 

f; \\A j+1 \\ j^ < \\C\\ \\B\\ e<IWI->' (7.9) 

is integrable, and in particular 

f'OO 

\ K(t)e~ at dt < oo 
Jo 

holds. Moreover, because of (7.9), the convergence 

N t j °° P 

3=0 J ' 3=0 J ' 

is dominated, for each fixed complex number s with Res > c, so we conclude 
using the formula 

C°° i' 

/ t 3 e~ st dt = -±- 

Jo ^ +1 

that 

/.oo oo 

/ K{t)er st dt = J2 A i s ~ J = W ^ ( 7 - 10 ) 

Jo j=1 

for all such s, when W is seen as a Taylor expansion about infinity. 

In general, consider any measurable matrix function F defined on [0, oo) and 
of exponential growth, by which we mean that ||.F|| is locally integrable and 

\\F(t)\\ e~ at -^Oast^ +oo 

for some u 6 1. For any such F, one defines its Laplace transform as the 
(matrix) function of a complex variable s, 



/*oo 

F(s):= / F{t)e" st dt. 
Jo 



This is well defined, and is analytic, for all s with Res > a, if a is as above. 
For instance, K is defined whenever a is larger than the real part of every 
eigenvalue of A. Observe that, writing W = q~ x P in rational form, we conclude 
from Equation (7.10) and Remark 6.7.5 that the matrix of rational functions 

H(s) := <?(s)- 1 P(s) 
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and K(s) coincide wherever they are both defined, and W is the Taylor expan- 
sion at infinity of this rational matrix. From now on we will not distinguish 
between H, W, and K(s). 

Since, for any two scalar functions of exponential growth, 

k * w = kuj , 

it follows, applying this formula to each coordinate, that for each input function 
uj of exponential growth the corresponding output 77 satisfies 

fi(s) = W(s)u>(s). 

For this reason one calls W the transfer matrix (or, in the particular case m = 
p = 1, the transfer function) of the system (A,B,C) (or of the behavior A): 
Its value at each "complex frequency" s characterizes the i/o behavior of the 
system at that frequency. In classical electrical engineering design, one often 
starts with a desired frequency behavior (for instance, the requirement that large 
frequencies be attenuated but low frequencies be relatively unaffected) and one 
builds a system in such a manner that the transfer matrix satisfies the desired 
properties. The following easy remark shows how each frequency is affected. 

Lemma 7.4.1 Assume that A is UBIBO. Pick any real number lu and any two 
integers I = 1, . . . ,p and j = 1, . . . , m. Consider the control 

u(t) = sincjf Vj 

(vj= canonical jih basis vector in M m ) on [0, 00). Let 77(f) be the Ith coordinate 
of the corresponding output on [0, 00), and write 

W{iw)ij = re iv 

in polar form. Then, 

r/(t) — rsm(u!t + ip) — > 

as f — > 00. 

Proof. Since the behavior is stable, K(t) is integrable, so 

\\K(t)\\ dr - 
Jt 

as t — * 00. Therefore, 



K^Vje*"**-^ dr 

t 



oc 



if(T)u je iw( *- r) dr - e iwt / K(T) Vj e- tuJT dr -> 0. 
Jo 

Thus, the imaginary part of this difference goes to zero, too, as desired. 
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Remark 7 '.4.2 Introducing more terminology from electrical engineering, for 
the case m = p = 1, one calls the modulus r of W{iuS) the gain at frequency to 
and its argument ip, taken in the interval [— 7r,7r), the phase shift at frequency 
to. Inputs at the corresponding frequencies are shifted and amplified according 
to these parameters. The system exhibits phase lag at a frequency if ip < 
and phase lead if ip > 0. The steady-state output corresponding to sinuit is 
rsm(u)t + ip). The Bode plot or Bode diagram consists of graphs of gain and 
phase as functions of lu > on a logarithmic scale. A large number of qualitative 
properties of the rational function W(s) can be deduced immediately from such 
plots, even on the basis of partial information. □ 

Exercise 7.4.3 Let m = p = 1. Assume that W(s) = jr^ is an asymptotically 
stable but unknown transfer function (a, b real > 0). Suppose that the steady- 
state output corresponding to the input sint is V2sin(£ — 7r/4). (That is, there 
is a —45° phase shift and an amplification of V2-) Find W(s). □ 

Exercise 7.4.4 Let m = p = 1. Show that if A is UBIBO, then the output 
corresponding to u = 1 converges to W(0). (The quantity 11^(0) is accordingly 
called the dc gain of the system, the gain corresponding to a nonalternating or 
"direct current" input.) □ 

Observe that, in general, if a control u> has exponential growth, then the 
solution of 

£ = A£ + Blo 

again has exponential growth, as is clear from the variation of parameters for- 
mula. Since £ is a linear combination of At; and Bui, it also has exponential 
growth, and an integration by parts shows that its transform is s£(s) — £(0). If 
£(0) = 0, we conclude that 

s£(s) = A£(s) + BCj 

and therefore that 

f] = C{sl - Ay 1 BCj 

for the output, consistent with the equality W = K . Laplace transform tech- 
niques are useful in developing a "calculus" for systems over K or C, though 
one could also use the formal power series approach of Section 6.7 to obtain 
analogous results (with the advantage that they then are valid over arbitrary 
fields of coefficients). 

Interconnections 

Consider two systems Si = (Ai, Bi,C\) and £2 = (^2,^2^2) and a linear 
interconnection as in Definition 7.2.3: 

xi = Atxi + B1CF11C1X1 + -Fi 2 C 2 a;2) 
x 2 = A 2 x 2 + B 2 (F 2 iCiXi + F 22 C 2 x 2 ) 
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seen as a system of dimension n\-\-rii and no inputs or outputs. (The connecting 
maps are fci = (^11,^12) and fc 2 = (-F21, -^22)-) 

We assume that £1 is controllable and observable, and we wish to study the 
possible linear systems £2 providing dynamic stabilization, that is, making the 
interconnection asymptotically stable. For each such possible system S 2 , we 
may consider a new interconnection in which F22 = 0, F21 = I, and F12 = I, 
since one can redefine A2 as A2 + B2F22C2, -B2 as -B2-F21, and C2 as -F12C2. 
Since a noncanonical system can always be replaced by a canonical one, we will 
look for canonical systems £2. Therefore, the problem becomes one of finding, 
for the given system Si, a canonical system E 2 and a matrix F so that 

±1 = Axxi + B 1 {FC 1 x 1 + C2X2) 
&2 = A 2 x 2 + B 2 C\Xi 

is asymptotically stable. For any such interconnection, we introduce external 
inputs as follows: 



xi = A\X\ + Bi(F(CiXi + U2) + C2X2 + Mi) 

±2 = A 2 X 2 + B2(CiXx + u 2 ) 



(7.11) 
(7.12) 



seen as a system of dimension n\ + n 2 , input space K m i+" l 2 ; and outputs 

C\Xx 



V 



C 2 x 2 



When the inputs are kept at zero and the outputs are ignored, this is the same 
as the above interconnection. 

Lemma/Exercise 7.4.5 If each of the systems Sj is canonical, then the system 
(7.11-7.12) is also canonical. □ 



y 2 



s V 



yi 



Figure 7.5: Interconnection, after simplifications. 



From this and the equivalences between the UBIBO property and internal 
stability, one concludes that the problem of characterizing the linear dynamic 
stabilizers of a controllable and observable system £1 is equivalent to the prob- 
lem of finding systems £2 and matrices F so that the interconection in the 
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present input/output sense is UBIBO. One then may work with the correspond- 
ing transfer matrices W\ of Si (of size p x m) and W 2 of £2 (of size m x p), as 
illustrated next. 

Consider the initial state (0, 0) of (7.11-7.12) and take any two inputs wi, U) 2 
of exponential growth. Let £1, £2 be the resulting trajectories, and let r\\ = C£i 
and r\i = C^ 2 be the corresponding outputs. Observe that 

fj 2 = W 2 e 2 , 

where 

e 2 ■= m + ^2, 

and 

rji = Wiei , 

where 

ei := -Fe 2 + 772 + Wi . 

Note that boundedness of r\\ and 772 is equivalent to boundedness of ei and e2, 
when wi and u>2 are bounded. It is more convenient to work with ei,e 2 rather 
than 771,772- We write W instead of W\, and we let 

V(s):=F + W 2 (s), 

thought of as a matrix of rational functions. Thus, the following equations are 
satisfied by the Laplace transforms of all signals: 

ei = o>i + Ve 2 

e 2 = Cj 2 + We\ ■ 

The matrix W is strictly proper, that is, its entries are rational functions van- 
ishing at infinity. On the other hand, V is in general only proper, that is, each 
of its entries can be written as p/q with the degree of p not greater than the 
degree of q. The product VW is again strictly proper, so as matrices of rational 
functions, 

/ - VW and I - WV 

are invertible (the determinant is 1 as \s\ — » 00). Thus, the above equations can 
be solved to obtain 

'ei\ __ w (&i 



e 2 I \uj 2 



where the "closed loop transfer matrix" W c \ is the proper matrix of rational 
functions (of size (777, + p) x (m + p)) 



(i-vw)- 1 (i-vwy^v 

(I-WV)~ l W (I-WV) 



-iw rr wm-i / • (7-13) 



We say that W c \ is stable if all of its entries are rational functions with no poles 
on Re s > 0. We have then: 
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Proposition 7.4.6 The canonical system £2 and feedback matrix F stabilize 
the above interconnection if and only if W c \ is stable. 

Proof. Stability of the interconnection is equivalent to the requirement that 
ei, e2 be bounded for all bounded wi, L02', we need to show that this is equivalent 
to stability of W c \. If the latter were strictly proper, the equivalence would follow 
from Proposition 7.3.5. We need only to see then that the proper case can be 
reduced to the strictly proper one. In general, every proper matrix U(s) can be 
written as U\(s) + Uq, where Uq is constant and U\ is strictly proper with the 
same poles as U. (To see this, one writes each entry as the sum of a constant 
and a proper rational function, after performing a polynomial division.) Let K 
be so that U\ is the transform of K. What we need to show is that the i/o 
behavior with impulse response K is UBIBO if and only if the operator norm 
of u) 1— > K * uj + Uquj is finite. But this norm differs from the one induced by 
convolution by K at most by the finite amount ||E/o||- ' 

Since every proper V can be written, after division, in the form F + W2, 
with Wi strictly proper, and in turn any such W2 can be realized as the transfer 
matrix of a canonical linear system, the problem of characterizing all stable 
interconnections can be restated simply as: 



Find all proper V so that W c \ is stable. 



The next section studies this problem for the case m = p = 1. 

7.5 Parametrization of Stabilizers 

We now specialize to the scalar case, in = p = 1, in order to avoid having to 
deal with the theory of stable factorizations of rational matrices. Most of what 
follows, however, can be developed in far more generality. 

To summarize, the algebraic problem that we need to solve is as follows: 
Given a strictly proper rational function w(s), find all those proper rational 
functions v(s) such that all three rational functions 

(l-vw)" 1 (1 - vw)~ 1 v (l-i™)" 1 ^ (7-14) 

are stable. Stability of a rational function w means that w is analytic on 

C+ := {s I Res > 0} , 

while properness is equivalent to the property that the limit 

w(oo) := lim w(s) 

\s\ — >oo 

exists (and is finite) and strict properness to w(oo) = 0. Our objective is to 
parametrize the set of all v stabilizing a given w. The most convenient way to 
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do this is through the study of the basic properties of the set RHqo of proper 
stable rational functions (the notation stands for real-rational Hardy space). We 
think of this set as a ring, when the pointwise operations for functions are used. 
The main technical fact that we need about this space is given in the following 
Lemma; it is a consequence of the somewhat more general fact — which we do 
not prove — that RHqo is a Euclidean domain. 

Two elements /, g G RHqo are said to be coprime if they have no common 
zeros in C+ and at least one of them is not strictly proper (in other words, / 
and g have no common zeros in C+ U {oo}). As with polynomials, coprimeness 
is equivalent to a "Bezout" type identity: 

Lemma 7.5.1 The functions /, g G RHqq are coprime if and only if there exist 
a, (3 € RHqo such that 

df + Pg = 1. (7.15) 

Proof. Since all functions are analytic on C+ and bounded at infinity, Equation 
(7.15) implies that / and g can have no common zeros in C+U{oo}. Conversely, 
assume that they are coprime. It is convenient to use the bilinear mapping 
z := (s — l)/(s + 1) sending C+ into the unit complex disk D and its inverse 
s := (1 + z)/(\ — z). Given any rational function w, we consider the rational 
functions 



w(z) 
w(s) 



1 + z 
\- z 
s- 1 

3+1 



Note that if w = W\W2 + w^ then w = W1W2 + W3, and similarly for". Further- 
more, w = w = w for all w, and w € RHqo •<=> w € S, where S is the set of 
all rational functions that have no poles in D. Coprimeness of /, g is equivalent 
to the requirement that / and g have no common zeros in D. The problem is 
then reduced to showing that, given any two a,b € S with no common zeros in 
D, there must exist a, (3 & S so that aa + f3b = 1. 

Write a = pi/qi and b = P2/02 as quotients of relatively prime polynomials 
so that p\ and pi have no common zeros in D. Let d be the GCD of the 
polynomials pi and find a linear combination d = a^pi + (3qP2 with polynomial 
coefficients. Note that d has no zeros in D because its zeros are the common 
zeros of p\ and pi ■ Therefore 

a := and p := 

d d 

are as desired. I 

By a coprime fractional representation for a rational function w we mean 
a pair of coprime f,g£ RHqo such that w = f/g. Any w admits such a 
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representation: If w = p/q as a quotient of relatively prime polynomials and if 
k is the maximum of the degrees of p and q, then we may write 

p/(s+l) k 
w — 



q/(s+l) k 

in coprime form. Observe that whenever w is strictly proper and w = f / g with 
f,g € RHqo necessarily / = gw is strictly proper, too. 

If w = f J ' g and v = n/d, then the conditions for stabilizability involving 
(7.14) can be restated as the requirement that all three elements 

^ H !± (7.16) 

r r r 

be in RH^,, where r := gd — fn. Observe that fn/r = gd/r — 1 is in RHqo if 
and only if gd/r is. Thus, fn/r can be added to the list (7.16). 

The main result is as follows. Given any w, we write it first in coprime form 
and find a, P as in Lemma 7.5.1. 

Theorem 34 Let w = f/g be strictly proper, and let a, (3 € RHoo be so that 
a f + Pg = 1- Then the rational functions v stabilizing w are precisely the 
elements of the form 



ij>9 



nf + P' 



/i£ RH C 



(7.17) 



Proof. Assume first that v is of this form. Since w is strictly proper, / is, too; 
thus, Pg = 1 — af implies that /3(oo) 7^ 0. We conclude that the denominator 
fif+P in (7.17) is nonzero, so the expression is well defined as a rational function, 
and in particular it is not zero at infinity so that v is proper. (Observe that v 
is in general not stable, however.) Consider the elements 

n := fig — a , d := \if + P 

of RHqo, and note that 

r = gd- fn = g{nf + p) - /(/iff - a) = af + j3g = 1 . 

Since their numerators are in RHqo, it follows that all of the elements of the 
list (7.16) are in RHqo, which proves that v defined by formula (7.17) indeed 
stabilizes. 

Conversely, assume that v stabilizes, and write v = n/d in coprime form. 
Let r := gd — fn; we claim that r is a unit in RH^, i.e., 1/r G RHqo. Indeed, 
if 7, S are so that ^n + Sd = 1, then 

1 1 , , , „ s , , , js ajfn aSfd fjjgn P5gd 
- = - (af + pg) (771 + da) = 1 1 1 

tyt iv* rv* iv* iv* iy* 
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is in RHqo, because each term is (by the assumption that v stabilizes). We 
may then redefine n := n/r and d := d/r, so that without loss of generality we 
assume from now on that gd — fn = 1 . We then have the equation 



a (3\ f\ _ (1 

-n d) \g) \l 

from which it follows, multiplying by the cofactor matrix, that 
7\ (d -P\ (l\ _ fd-P' 



(7.18) 



1 g I \n a J V 1 J \n+a 

where /z = ad+(3n G RHqo is the determinant of the matrix in (7.18). Therefore 
d = [if + (3 and n = fig — a, as wanted. I 

Example 7.5.2 As a trivial illustration, take the system x = u, y = x. Its 
transfer function is w = 1/s, which admits the coprime fractional representation 

w= 1 /( 5+1 ) 
s/(s + l)' 

and we may pick a = j3 = 1. According to the Theorem, every possible stabilizer 

is of the form 

fis/(s + 1) — 1 flS — s — 1 

~ li/(s + l) + l ~ fi + s+1 ' 

with fi ranging over all possible /! € RHqo. Such v's are guaranteed to be proper 
and to stabilize. For instance, for fi = one obtains v = — 1, which corresponds 
to the obvious stabilizing law u = —y. □ 

The usefulness of Theorem 34 is in the fact that it characterized all stabilizers 
in terms of a free parameter. Once such a parametrization is obtained, further 
design objectives can be translated into requirements on the parameter. In 
this sense it is useful to note that the closed-loop transfer matrix can also be 
computed easily; with the notations of the Theorem, one has 



f gd gn\ (fg g 2 \ ( g(3 -ag 

\fd gd ~ fI \p fg + 1//3 gp 



which is an affine expression fiWi + Wq on fi. 

Exercise 7.5.3 With the above notations, show that, for any desired closed- 
loop transfer matrix W = W c \, there will be a solution if and only if one can 
write 

with ag 2 = bfg, and both of these rational functions are in RHqo. □ 
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Exercise 7.5.4 Prove that, in general, /i is uniquely determined by v. That is, 
each possible v appears exactly once in the parametrization. □ 

Exercise 7.5.5 It is not totally trivial to find the coefficients a and j3 as in 
Equation (7.15). Show how to obtain these coefficients starting with any partic- 
ular stabilizer. (A stabilizer can be itself obtained from state space techniques, 
for instance.) □ 

Exercise 7.5.6 Find all stabilizers for w = 1/s 2 . Are there any that are con- 
stant? □ 

Nyquist Criterion 

There is a graphical test for stability that is standard in engineering practice 
and which helps in finding stabilizers. Again for simplicity of exposition we 
restrict to the case m = p = 1 and to stabilization by constant feedback. Before 
presenting the main result, we need to review some basic complex analysis facts. 
Assume that / is a rational function having no poles or zeros on a simple 
closed curve C. Assume in addition that C is oriented clockwise (to respect 
engineering conventions). Then 

2™ Jc f{s) 

where Zq (respectively, Pq) is the number of zeros (respectively, poles) of / in 
the region bounded by C (counted with multiplicities). Consider also the curve 
r := /(C), with the induced orientation. Changing variables, the integral (7.19) 
is the same as 

J_ f # 

In turn, this last integral is the index of T with respect to the origin, that is, 
the number of counterclockwise encirclements of by the oriented curve I\ We 
let cw (r,p) denote the number of clockwise encirclements of the point p by the 
curve r. Therefore, cw (r, 0) = Zc — Pc- 

Now assume that / is proper and has no poles or zeros on the imaginary axis 
iM.. Let r be the image of the curve iM. U {oo}, seen as an oriented curve in the 
Riemann sphere (transversed on the imaginary axis from negative to positive 
imaginary part). Let P + and Z + be the numbers of poles and of zeros of / on 
the half-plane C+; then 

cw (L, 0) = Z+-P+. (7.20) 

(We could also argue directly in terms of the complex plane: The curve Y is the 
closure of f(iM), which is the same as the union of f(iM) and the point /(oo), 
seen as an oriented curve, with the orientation induced by transversing iM. from 
— ioo to +ioo. The numbers P + and Z + are equal to the respective numbers in 
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the region bounded by the curve C r , for all large enough r, where C r consists of 
the interval [—ir, ir] of the imaginary axis and the semicircle of radius r centered 
at the origin and on the right-hand plane (see Figure 7.6); on the other hand, 
the image f(C r ) approaches the image of iR, since all values on the semicircle 
approach /(oo), so the index is the same.) 




Figure 7.6: The path C r . 

We are ready to apply these considerations to the stabilization problem. We 
will deal with stabilization by constant feedback, that is, v = k is a constant 
rational function. 

Lemma/Exercise 7.5.7 Assume that w is rational and that v = k ^ is a 
constant. Then the poles of the entries of W c \ (that is, the elements in the list 
(7.14),) are precisely the zeros of w — -r. □ 



We call the zeros of ' 



j- that are in C + the unstable closed-loop poles (for 
the system that results when using v = k as a stabilizer) and the poles of w in 
C+ the unstable open-loop poles. Thus, v = k stabilizes w if and only if there 
are no unstable closed-loop poles. 

Let w be a strictly proper rational function. We assume that w has no poles 
in «R (the general case can be treated similarly, simply by modifying the curve 
using a detour about the poles). Denote by T the closure of the image of iM. 
(transversed from — ioo to -Moo) under w. The following result is often referred 
to as the Nyquist criterion: 

Theorem 35 For each k ^ such that \ ^ T, 



cw(r, 



number of unstable closed-loop poles 
— number of unstable open-loop poles 



Proof. First note that cw (r, jr) = cw(Ffc,0), where T^ is the image of il 



under f(s) 



-r (since Tk is just a translate of T). Note that / has no zeros 



or poles on iR, by the assumptions on w and k. Since / is proper, Equation 
(7.20) applies to Tk- ■ 
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Figure 7.7: Nyquist diagram for Example 7.5.9. 



Corollary 7.5.8 If w is stable, then for each k ^ such that \ ^ T, the 



closed-loop system with v = k is stable if and only if cw (I\ i;) 



0. 



□ 



Observe that when ^ 6T necessarily w — -r has a zero in jR, so the closed- 
loop system is unstable. And for k = one has the same stability as w. Thus, 
the above criterion and its corollary completely determine the range of fc's for 
which stability holds. To apply these results, one plots w{iuj),uj G R + and then 
one obtains T by symmetry (since w is real, w(— iu>) = w(iu>)). Unless w = 0, 
there will be a finite number of points where the curve T crosses the real axis. 
These points determine a finite set of open intervals in each of which the index 
is constant. For each such interval (a, 6), the condition i G (a, 6) provides a 
characterization of the number of closed-loop poles when using the feedback law 
U = ky. 

Example 7.5.9 Take the transfer function 

n s-1 

w(s) 



(s+l) 2 (s + 2) 2 ' 

which is clearly stable. A computer plot reveals that the index is constant on 
each of the intervals (approximately) 

(-oo,-l/4) (-1/4,-0.1034) (-0.1034,0) (0,0.14923) (0.14923, +oo) 
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(see Figure 7.7), where it is 0,1,3,2,0, respectively. We conclude that —4 < 
k < 6.7 gives the range of stability, and that there are 3 unstable closed-loop 
poles when k < —9.671, 1 when k G (—9.671,-4), and two such poles when 
k > 6.7. □ 



7.6 Notes and Comments 

Observers and Detectability 

Asymptotic observability is obviously a necessary condition that must be satis- 
fied if there is any hope of even asymptotically final-state distinguishing the state 
a; from other states. Any algorithm that uses only the output measurements 
will not be able to differentiate between x° and states in X° if the input hap- 
pens to be constanly equal to u . So unless these states already converge to x°, 
there is no possibility of consistent estimation. For linear systems, detectability 
as defined here turns out to be equivalent to the existence of observers, but in 
the general nonlinear case a stronger definition is required (see, for instance, 
[361]). Observers can be seen as a deterministic version of Kalman filters, intro- 
duced originally in [214] and [229]; the deterministic treatment seems to have 
originated with work of J.E. Bertram and R.W. Bass, but the first systematic 
treatment was due to Luenberger in the early 1960s; see [287] for references and 
more details. The construction of reduced order observers in Exercise 7.1.5 is 
taken from [287], who followed [159]. 

The terminology "strong" in Definition 7.1.3 is meant to emphasize that 
the observer must provide estimates for all possible controls, and starting at 
arbitrary initial states. It is possible — and even desirable — to weaken this 
definition in various ways, by dropping these requirements. For linear systems, 
all reasonable definitions (even requiring that there be some initial state for the 
observer that results in converging estimates when the zero control is applied to 
the original system) are equivalent in the sense that the existence of an observer 
of one type implies the existence of observers of the other types. For nonlinear 
systems, these different notions are of course not equivalent; see, for instance, 
[87]. Some references on observer construction for nonlinear systems are [87], 
[164], [209], and [253]. There are technical reasons for not making the definition 
slightly more general and allowing 9 to depend on u>(t): This would mean that 
the instantaneous value of £(£) may not be well defined when dealing with con- 
tinuous-time systems, as there u is in general an arbitrary measurable function. 

Dynamic Feedback 

The design of dynamic controllers using a combination of an observer and a state 
feedback law is a classical approach. It turns out that even optimal solutions, 
when the (linear quadratic) problem is posed in the context of stochastic con- 
trol, also have this separation property. For nonlinear stabilization using such 
techniques, see, for instance, [416] and [419]. For linear systems over rings, see, 
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for instance, [129] and [246]; for infinite dimensional linear systems, see [201] 
and the references given therein. 

External Stability for Linear Systems 

The topic of external stability for linear infinite dimensional and for nonlinear 
systems has been the subject of much research. The books [115], [414], and 
[422] provide further results and references, and [410] was an influential early 
paper. 

The notion of input-to-state stability (ISS) has recently become central to 
the analysis of nonlinear systems in the presence of disturbances. The definition 
is from [370]; for recent results see [376]. In the latter reference, it is shown that 
a system is ISS if and only if it is internally stable, complete, and has a finite 
nonlinear gain. The textbook [259] discusses at length applications of ISS and 
related properties to practical nonlinear feedback design problems. The "dual" 
notion of OSS is given in [377], where a generalization to systems with inputs as 
well as outputs ("input/output-to-state stability" or IOSS) is also considered. 

For questions of i/o stability versus stability in a state-space setting for 
infinite dimensional linear systems see, e.g., [285], [324], and [439]. 

Frequency Domain Considerations 

There is a very large number of textbooks dealing with complex variables tech- 
niques for the analysis of linear systems. A modern presentation of some of these 
facts, together with references to a bibliography too extensive to even attempt 
to partially list, can be found in [144]. 

Parametrization of Stabilizers 

See, for instance, [417] and [145] for detailed developments of the theory of 
parametrizations of stabilizing compensators. Theorem 34 is due to Youla and 
coworkers, and was given in [441]. In its version for arbitrary m,p it is the ba- 
sis of many of the recent developments in multivariable linear control. Recently 
there have been attempts at generalizing such parametrizations to nonlinear sys- 
tems as well as obtaining the associated coprime factorizations; see, for instance, 
[116], [171], [172], [258], [370], [398], and [412]. 
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Chapter 8 

Optimality: Value Function 



Chapter 3 dealt with the abstract property of controllability, the possibility of 
using inputs in order to force a system to change from one state to another. 
Often, however, one is interested not merely in effecting such a transition, but 
in doing so in a "best" possible manner. One example of this was provided in 
that chapter, when we calculated the control of minimal norm that achieved a 
given transfer, in a fixed time interval. 

The general optimal control problem concerns the minimization of some 
function 

of trajectories, the cost function or performance index. (Or, one may want 
to maximize instead a "utility" function F, but this amounts to minimizing 
the cost —F.) The performance index F might incorporate energy costs, or 
the length of time that a control takes to perform a desired objective, and the 
optimization may be subject as well to constraints such as maximum allowed 
magnitudes of controls. Of course, £ and lu must be related by the system's 
equation, for instance 

in continuous-time. 

In broad terms, it is possible to classify the theoretical study of optimal 
control into two major approaches: dynamic programming, and variational 
methods. (We include in this second category the Pontryagin Maximum, or 
"Minimum", Principle.) Both approaches rely upon a transformation of a dy- 
namic optimization into a set of static optimization problems, one for each time 
instant, together with a differential (or difference) equation. This equation in- 
volves the state, as well as an auxiliary quantity which serves to link the static 
problems. In dynamic programming, the auxiliary information is encoded into 
an instantaneous cost associated to a current state. In variational approaches, 
one works, instead, with a Lagrange multiplier. 

347 
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In dynamic programming , the central role is played by the value function 
or Bellman function (named after Richard Bellman, who pioneered the use of 
the technique). This is the function V(t,x) that provides, for any given state 
X at any given time t, the smallest possible cost among all possible trajectories 
starting at this event. The main observation is that along any optimal trajec- 
tory it must hold that the initial cost plus the cost of a single transition add up 
to the remaining optimal cost. In discrete-time, there results a recursive, easily 
programmed method for optimization, with a multi-stage optimization reduced 
to a sequence of single-stage minimizations. In continuous-time, the recursion 
becomes a differential equation in time, that is to say, a partial differential 
equation for V{t,x), called the Hamilton- J acobi- Bellman equation (HJB). In 
general, this PDE is very hard to solve, and no solutions exist unless one allows 
a generalized notion of solution. In the case of linear systems with quadratic 
costs, the value function is quadratic in x, and because of this, the HJB equa- 
tion reduces to an ordinary differential equation, the Riccati matrix differential 
equation, which can be readily solved numerically. 

In variational or multiplier methods (in modern control theory, Pontryagin's 
Maximum, or Minimum, Principle (PMP)), one introduces a function p = p(t), 
as a "Lagrange multiplier" for the optimization problem, and, roughly speaking, 
studies the critical points of (in continuous-time): 

F(S,w) + (p,£-f(S,w)), 

seen as a function of the state £(•), the control lu(-), and the multiplier p{). (In 
the Hamiltonian formulation of classical mechanics, p is the vector of generalized 
momenta.) Setting derivatives to zero, one may in principle solve for u)(t) as a 
function of x(t) and p(t), and in addition one obtains a differential equation for 
the pair of vector functions (£,p). 

Each of the two approaches has its advantages and disadvantages. Dynamic 
programming-based techniques, when applicable, often result in solutions in 
feedback form, which is highly desirable in applications, but the most general 
theorems tend to state only sufficiency conditions for optimality of a proposed 
solution. On the other hand, variational techniques tend to require less assump- 
tions about the system and are thus more generally applicable, but they lead 
only to necessary conditions. Variational methods are the subject of Chapter 9. 

The main objective of this chapter is to discuss the dynamic programming, 
or value function, approach, and in particular the application to linear systems 
with quadratic costs ("LQ problems"). For LQ problems, both methods lead 
to the same solution, but the discussion in terms of dynamic programming is 
somewhat easier. The first section discusses dynamic programming in general 
abstract terms. That section may be skipped, and the reader may proceed 
directly to Section 8.2, which discusses linear systems. However, the optional 
section explains the general context in which to understand the linear result, 
in particular the Riccati matrix differential equation. After that, we treat the 
infinite-time (steady-state) LQ problem. The chapter closes with an infinite- 
horizon problem stated in nonlinear terms. 
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As a final introductory remark, we note that the study of optimal control 
theory is important for reasons other than optimality itself. For nonlinear sys- 
tems, it is often difficult to find any controls whatsoever that help achieve a 
desired transfer. Optimal control offers a systematic approach to search for 
these controls, an approach that can be used as the basis of numerical methods. 

8.1 Dynamic Programming* 

We first make some general remarks about abstract optimization problems. 
Given a system £ and a function taking nonnegative real values 

Q : V^ -» M+ , 

which we shall call the trajectory cost function, we read 

Q(t,o-,x,oj) 

as "the cost of the trajectory produced by the control u> € 'W-' T,T > if we start at 
state x." The following additivity property is assumed to hold for Q, for each 
triple of times t\ < £2 < ^3, and each (x,uj) G D tl ,t 3 '- 

Q(t 3 ,tl,X,U)) = Q(t2,tl,X,Uj\[ tu t 2 ))+ Q(t3,t2,(/}(t2,tl,X,U)\[ tut2 )),Uj\[t 2 ,t 3 )) 

(8.1) 
so that in particular it holds for the empty control that 

Q{t,t,x,o) = 

for all t and x. 

The main examples are as follows. If £ is a discrete-time system 

x + = V(t,x,u) , 

then for any given function 

? : T x 1 x U ^ R + (8.2) 

(with T = Z) we associate the trajectory cost Q defined by Q(t,t, -,o) = and 



Q(T,a,X,U>) := ^ q{i, (,{i), uj{ij), 



where £ := ip(x,Lu). If instead £ is a continuous-time system 

x = f(x,u) 



* This section can be skipped with no loss of continuity. 
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and there is given a function (8.2) (with T = K), we may consider the trajectory 
cost 

Q(t, a, x, w) := / q(s,£(s), w(s)) ds , 

where £ is as above. In order for this expression to be well defined we may 
assume for instance that q(t, x, u) is continuous on all of its arguments. We 
refer to the function q, in both the continuous-time and discrete-time cases, as 
the instantaneous cost. 

Assume that in addition to the trajectory cost Q there is also given a function 

p : X -> R + , 

which we will call the terminal cost function. Then we define on D^, 

J(t,<t,x,u) := Q{t,o-,x,uj)+p{Z{t)), 

where £ = ip(x,u>), the total cost function. 
The problem we wish to study is: 

Given a system E, a trajectory cost function Q and a final cost 
function p, a pair of times a < r, and a state x°, find a control 
u> € W (7 ' T ' admissible for x° which minimizes J(r, (J, x°,uj). 

More precisely we want to study existence and uniqueness of, as well as algo- 
rithms for constructing, such controls. We call the data 



E,Q,p,a,r (OCD) 



a set of optimal control data. When an initial state x° is given together with 
(OCD), one has an optimal control problem. (As discussed earlier, more general 
problems can also be studied, for instance by adding final state constraints. In 
this chapter, we only deal with problems of the above type, however.) 

Dynamic Programming 

The dynamic programming approach is easier to understand in discrete-time, 
even though it applies equally well for continuous-time systems. In particular, 
it is worth describing first how it arises in the purely combinatorial case of 
automata theory. 

Finite Systems 

Assume that both X and li are finite in problem (OCD), and that E is a dis- 
crete-time system. Denote by 

N :=card(X), M := card (IX) 
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the cardinalities of X and IX, and let 

k := r — a . 

For simplicity of notations, we assume that E is complete, that is, the local 
transitions V(t, x, u) are defined for all possible (t, x, u). If this is not the case, 
one can modify the arguments to follow in an obvious manner in order to avoid 
undefined transitions. 

The most naive attack on the optimization problem would consist of an 
enumeration of all possible controls u> £ VS a ' T \ thought of as sequences 

Ul,...,Uk 

of elements of IX, followed by the calculation of the complete trajectory (k steps) 
corresponding to such a control starting at initial state x , and the computation 
of its associated cost. Since there are M k such sequences, this results in the 
order of 

kM k 

evaluations of V and of the instantaneous cost. (Actually, slightly more com- 
putation is needed for the calculation of the final cost and for the comparisons 
needed to find the minimum among all resulting trajectories.) 

Alternatively, one could argue as follows. We shall construct inductively in 
time, backward from r toward a , two functions 

V : [a, t] x X -> R + 

and 

K : [a, t) x X -> IX 

called, respectively, the Bellman function and an optimal feedback law, such that 
the following properties hold: 

1. For each s € [a, r] and each x € X, 

V(s, x) = min J(t, s, x, u>) . (8-3) 

2. For each s € [o~,t) and each x £ X, if £ is the solution on [s,t] of the 
difference equation 

£(Z + 1) = P(l,S(l),K(l,Z(l))) 1 = 8,8+1,. ..,t-1 (8.4) 
£(s) = x 
(the closed-loop equation) and we define 

v(l):=K(l,H(l)), (8.5) 

then this control solves the optimization problem starting at (s,x), that 
is, 

V{s,x) = J{t,s,x,v). (8.6) 
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Observe that once these functions are obtained, the original problem 

min i7(r, a, x ,u>) 

CO 

is also solved, but in the process one also obtained solutions for all other initial 
states as well, a characteristic that is useful in applications, as discussed later. 
We now show how to inductively obtain V and K . We start by defining the 
boundary value 

V{t,x) :=p(x), 

which obviously satisfies (8.3). Assume now that V already has been defined 
for all (s,x) such that 

t < S < T , 

and K has been defined for t < s < r, in such a way that the above properties 
hold. We now extend the definition to all pairs of the form (t,x), as follows. 
Given any such (t, x) compute the minimum 

V(t,x) := mm[q(t,x,u) + V(t + l,T(t,x,u))] (8.7) 

and let K(t, x) be defined as any u that minimizes this expression. 

We must establish that with these definitions V and K satisfy the above 
properties. Let uj € \l [t,T > be optimal from (t,x), 

J(t, t,x, coo) = min J(t, t,x,w) . (8.8) 

By cost additivity, 

J(t, i, x, uj ) = q(t, x, u ) + J{t, t + l,V(t, x, U ),u>i) , (8.9) 

where we are denoting u := cj (£) and u)\:= restriction of ujq to the interval 
[t + 1, r). The dynamic programming principle states that u)\ is optimal for the 
problem after time t + 1, i.e., 

J{T,t + l,V{t,x,u ),ui) = V{t+l,V(t,x,u )). (8.10) 

This is proved by contradiction: If false, then there is some v so that 

J(t, t + l,V(t, x, u ),v) < J{t, t + 1, P{t, x, u ), wi) 

from which it follows that the concatenation u> € W t,T > of uo and v has cost 

J(t, t, x, u>) = q(t, x, u ) + J(t, t + l,V(t, x, Uo),v) < J{t, t, x, uj ) , 

which contradicts (8.8). We conclude from (8.7), (8.9), and (8.10) that 

V(t, x) < min J{t, t, x, u>) . 
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To prove equality it will be sufficient to prove that, if we solve (8.4)-(8.5) with 
initial time t, then (8.6) holds. 

Let v be as in (8.5); we write it as the concatenation of uq := K(t,x) and 
some oji G U\ t+1 ' T '. By inductive hypothesis, since uj\ is defined by the closed- 
loop equation starting at V(t,x,Uo), it follows that u>\ is optimal, that is, (8.10) 
holds. Therefore 

J(t, t, x, v) = q(t, x, m ) + V(t + l,V{t, x, uo)) • 

On the other hand, u® = Kit, x) was defined by the property 

q{t,x 7 u ) + V{t+ l,V(t,X, «o)) = mm[q{t,x,u) + V(t + l,V{t,x,u))] 

uGli 

= V(t,x), 

so (8.6) holds, as desired. I 

It is instructive to compare the computational effort involved in this solution 
method with that in the naive approach. A larger amount of storage may be 
required for intermediate results since, unless a "general formula" is obtained for 
V and K, all values must be tabulated. We ignore this (extremely important) 
aspect and concentrate on a rough count of the number of operations needed. 

Basically, one must perform for each t G [er, t) and each x G X the mini- 
mization in equation (8.7). This takes of the order of M = card (11) operations, 
and it must be done kN times. Finally, a forward pass to solve the closed-loop 
equation is necessary, which takes an additional O(k) operations. In all, the 
number of operations is 

0{kNM) 



operations as opposed to the 



0{kM k ) 



needed in the enumerative approach. (If the problem had been to solve for 
all initial states x, then this expression would instead be 0(kNM k ).) One 
typical situation is that in which continuous state and input spaces K™ and 
K m are quantized by using h values in each coordinate, and the transition map 
is defined on the discretized values by evaluating V at a midpoint and then 
truncating the result. In this case N = h n and M = h m , so the estimates are 

0(kh mk ) and 0(kh m+n ), 

Often in and n may be small but k and h are large, in which case the advantage 
of the second (dynamic programming) approach is evident. However, since the 
second expression is exponential in n and m, it is clear that for large norm the 
computational effort required even by dynamic programming may be prohibitive 
(the "curse of dimensionality" ) . 

Example 8.1.1 Sometimes the minimum in equation (8.7) can be computed 
in closed form, in which case dynamic programming does provide an especially 
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useful technique. For instance, consider the problem of minimizing the quadratic 
cost 

E -(0 2 + C(r) 2 

l=a 

for the one-dimensional linear time-invariant system 

x + = ax + u . 

Such linear systems with quadratic cost are the subject of this chapter, and a 
general solution will be given later. But it is useful to compute explicitly with 
one example. Here q(t,x,u) = u 2 and p(x) = x 2 , so we must solve the partial 
difference equation 

V(t, x) = min{w 2 + V{t + l,ax + u)} (8.11) 

u 

with boundary condition V(r, x) = x 2 . At t = r the function V is quadratic in 
x, and inductively it is clear from (8.11) that, if V(t + 1, •) is quadratic in x, 
then V(t, •) also is. So we look for a sequence of numbers nt such that 

V(t, x) = TT t X 

satisfies (8.11). Minimizing 

U + ir(ax + u) 
as a function of u gives a unique minimum of 



1 +7T 

at 

IT 

u = ax . 

1 +7T 

Thus, there is a simple recursion 

""t+i 2 1 

■K t := — a , 7r T := 1 , 

1 + TTi+l 

and the feedback law 

isn- \ 7r *+ 1 
K{t,x) := ai 

1 + TTt+l 

provides the desired optimal control. □ 

One advantage of dynamic programming over other optimization techniques 
is that it provides automatically a feedback law. In actual control implemen- 
tations, this feedback law is stored, rather than the optimal control sequence 
obtained in the forward pass of the algorithm (solving (8.4)). The practical 
benefits of feedback control were discussed in Chapter 1. Another advantage of 
dynamic programming is that it is easy to modify it to account for state and 
control constraints. 
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Exercise 8.1.2 Consider the discrete-time time-invariant one dimensional sys- 
tem with X = R, transitions 

X + = X + u , 

and control value space the nonnegative reals: 

U = R+. 

With q(t,x,u) := u 2 , p(x) := x 2 , a = 0, and r = 3, find V{t,x) for all x and 
t = 0,1,2 using the dynamic programming technique. Next guess a general 
formula for arbitrary t, a, T and establish its validity. □ 

Exercise 8.1.3 Develop an approach to the general discrete-time problem, say 
for simplicity with finite X and 11, which relies on a forward rather than backward 
first pass. To each event (£, x) you will have to associate a "cost until now" 
together with a pointer to a triple (t — l,z,u) for which V(t — l,z,u) = x. 
Observe that this solution technique does not provide a feedback law for the 
system. □ 

The General Principle 

The dynamic programming ideas in principle can be applied to arbitrary sys- 
tems. We develop here the basic facts. 

Definition 8.1.4 LetT,,Q,p,a,T be a set of optimal control data (OCD). The 
Bellman function (or value function, or optimal cost-to-goj associated 
to this data is the function 

V(t, x) := inf {J(t, t, x, u) \ (r, t, x, w) G V^} (8.12) 

defined on [a, r] x X and with values in M + U {+00} . □ 

We make the convention that V(t, x) = +00 when the set of controls on [t, r) 
admissible for x is empty. Note that V(r, x) = p(x) for all x. 

The finite problem studied earlier had the following desirable property: 



For each (t, x) there is a control u) so that V(t, x) = J(t, t, x, u>) 



In other words, the "inf" in (8.12) is in fact achieved as a minimum. In order not 
to have to repeat the statement of this property, we shall say that the Bellman 
function of (OCD) is achieved if this happens. A control u) for which the above 
property holds will be called optimal (for the event (t,x)). The corresponding 
£ = i/j(x,lo) is an optimal path and (£, u>) is an optimal trajectory. 

Lemma 8.1.5 The following properties hold for any data (OCD), for each s,t 
such that a < t < s < r, and each iGl: 

1. For all lu so that (s,t, x, ui) € 2?^,, 

V(t,x) < Q(s,t,x,oj) + V(s,(j){s,t,x,u>)). (8.13) 
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2. If the Bellman function is achieved, then 

V(t,x) = Q(8,t,x,u\lt,.))+V(s,(l>(8,t,xMlt,.))) (8- 14 ) 

for any u> optimal for (t, x). 
It follows that 

V{t,x) =mm{Q(s,t,x,w) + V(s,(t>(s,t,x,u)))}, (8.15) 

where the minimum is taken over all u> so that (s, t, x, u>) G T>^. 

Proof. Assume that s, t, x,u> are given. Pick any e > 0. We prove that 

V(t,x) < Q(s,t,x,u)) + V(s,(t>(s,t,x,u))+e. (8.16) 

Since e is arbitrary, (8.13) will follow. By definition of V(s,<j>(s,t,x,U))), there 
exists some control v on [s,t) so that 

J(r,s,(t>(s,t,x,u)),v) < V(s,<f>(s,t,x,u))) + e 

(if V(s, 0(s, i, x, w)) = oo, then there is nothing to prove, so we may assume that 
this is finite). Let u' be the concatenation of u> and v. Since a/ is admissible 
for the state x, it follows by definition of V(t, x) that 

V(t, x) < J{t 7 t, x, u/) = Q{s 7 1, x, w) + J"(r, s, aj, f ) , 

and this gives the inequality (8.16). 

We now consider the second part of the Lemma. Let ui be optimal for (t, x). 
The equality (8.14) holds, by additivity of the cost, if the last term is instead 

J{T,8,4>{8,t,X,w\[t,8)),w\[8,T))> 

so it is only necessary to see that the restriction of u> to [s, r) must be optimal. 
This is precisely the conclusion of the dynamic programming principle, as in the 
discrete case, since if this restriction were not to be optimal, then the concate- 
nation of some other control on [s,t) with its first part w|r t)S ) would provide a 
control for (t, x) with less cost. I 

The next Lemma states essentially that the above properties characterize the 
Bellman function; it is a verification principle in the sense that, if a function V 
and controls lu are shown to satisfy the Lemma, then one has verified optimality. 

Lemma 8.1.6 Given a set of optimization data (OCD), assume that 

V : [a, t] x X -» R 

is any function so that the following properties hold: 
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1. For each s, t such that a < t < s < r and each i£l, 

V(t,x) < Q(s,t,x,u) + V(s,<j){s,t,x,u)) (8.17) 

for all to on [t, s) admissible for x; 

2. for each s, t such that a < t < s < t and each x G X there exists some 
control Z) such that 

V(t,x) = Q(s,t,x,uj) + V{s,<f>{s,t,x,u)) ; (8.18) 

and 

3. the boundary condition 

V(t, x) = p{x) (8.19) 

holds for all x. 
Then it must hold that the Bellman function is achieved and V = V . 

Proof. Fix any t, x. As a particular case of the first property and using the 
boundary condition one concludes for any admissible w on [t, r) that 

V(t,x) < Q(r,t,x,u>) +p((f>(T,t,x,u>)) = J(r,t,x,u>) . 

By the second property and an analogous argument there is at least some lu so 
that 

V(t,x) = J(T,t,X,w), 

so the function V must be the Bellman function for the problem, and the ex- 
istence of the controls uj implies that this is achieved (with the respective w's 
being optimal). I 

The Continuous-Time Case 

Let us turn now to a continuous-time system £ (Definition 2.6.4 in Chapter 2) 

x= f(t,x,u) 

over IK = K for which / is assumed to be continuous on t. First we will outline 
the derivations; rigorous proofs are given later. 

We assume given optimal control data (OCD) for this system, where 

q : R x X x It -> M.+ and p : X -> R+ 

are both continuously differentiable. These give rise, as above, to the perfor- 
mance criterion or cost 

J(t, a, x, w) := / q{v,£(v),u(r)))dri + p(${r)) 
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to be minimized for any given x, where £ = ip(x,u>). 

The conclusions of Lemma 8.1.5 provide infinitesimal information about the 
variation of V, as follows. Equation (8.14) says that, for each optimal control 
for (t,x), 



V(t,x)=J q(T),£(v)Mv))dt) + V(8,£(a)) 

for all s. Thus, the Bellman function evaluated along an optimal trajectory, 

is absolutely continuous and has derivative 



ds 



-q{s,£{s),u>(s)) 



almost everywhere. That is, along an optimal trajectory the instantaneous de- 
crease in the Bellman function is equal to (minus) the instantaneous cost. 

Because dV(s,^(s))/ds = V s + V x t;, we can also write this equation (using 
the argument t instead of s): 

V t {t,x) = -q(t,x,u>(t)) - V x {t,x)f{t,x,uj{t)) , (8.20) 

provided that the partial derivatives make sense. (In interpreting the equation, 
note that the gradient V x of V with respect to i is a row vector.) Along 
arbitrary trajectories, equation (8.13) suggests that the Bellman function will 
decrease less fast than the instantaneous cost, giving the inequality 

V t (t, x) > -q(t, x, i/(i)) - V x (t, x)f{t, x, i/(t)) (8.21) 

for every other (not necessarily optimal) control v. (Both sides of (8.13) are 
equal at s = t, so at least for small enough s — t one expects that the derivatives 
with respect to s will satisfy the same inequality, implying (8.21); this is made 
precise in the proof of the Proposition below.) Note that only the value of the 
control at the given time t appears in this inequality and in the equality (8.20). 
If the Bellman function is achieved and if there is enough smoothness, both of 
(8.20) and (8.21), together with the natural boundary condition on V, can be 
summarized by the equation: 



V t {t, x) = - mm ueU {q(t, x, u) + V x (t, x)f(t, x, u)},V(t, x) = p(x) (HJB) 



called the Hamilton-Jacobi-Bellman equation associated to the optimal con- 
trol data. 

The significance of (HJB) lies in its reducing the problem to an optimization 
at each stage (find u that minimizes, for each fixed (t,x), and solve the partial 
differential equation for V), in a manner totally analogous to the discrete-time 
case. 
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Note that in the discrete-time case one has instead the following discrete 
HJB equation or dynamic programming equation: 



V(t, x) = mm u&u {q(t, x, u) + V(t + l,V(t, x, «))}, V(t, x) = p{x) (DPE) 



as a specialization of (8.15). Observe that (DPE) is an equation that evolves 
backward in time; for differential equations this time reversal is reflected in the 
negative sign in Equation (HJB). 

Definition 8.1.7 Let (OCD) be optimal control data for a continuous-time sys- 
tem. We say that there is a smooth solution (of the optimal control problem 
associated to this data) if the following properties are satisfied: 

• The Bellman function V is of class C 1 for (t,x) G [it, t] x X, and 

• for each event (t,x) G [er, r] X X there is a continuous control u) such that 

V(t,x) = J(T,t,X,u). 

In particular, the Bellman function is achieved. □ 

The main facts are summarized by the next two results. 

Proposition 8.1.8 Assume that there is a smooth solution of the problem 
(OCD) on [a,r]. Then, 

(i) The Bellman function V satisfies the (HJB) equation for all (£, x) G (a, r) x 
X; and 

(ii) if (£,<^) is any optimal trajectory and la is continuous, then at all pairs 
(t,x) = (£,£(£)) along this trajectory, the minimum in (HJB) is attained 
at w{t). 

Proof. The final condition V(t, x) = p{x) holds by definition of V . Equation 
(8.20) holds at each (t,x) along any optimal trajectory corresponding to a con- 
tinuous control, because of the assumed differentiability of V and the preceding 
arguments. Therefore, in order to establish (ii) it is only necessary to show that 

V t (t, x) + V x {t, x)f(t, x, u) + q(t, x,u)>0 (8.22) 

for every control value u, at any given (t, x). Consider any such u and let cu e be 
the control with constant value u on the interval [t,t + e\. For small enough s 
this is admissible for x, and by (8.13), 

rt+s 

g (e) :=V(t + e,cf>(t + e,t,x,u> E ))-V(t,x)+ / q(t),S(v), «) dq > 

along £ = tp(x,u> s ) for every e > 0. It follows that g'(0) > 0, which implies 

(8.22). 
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To prove (i), it is only needed to show that every possible (t, x) G (c, r) x 
X appears as part of an optimal trajectory. But this fact is an immediate 
consequence of the assumption that the problem (OCD) has a smooth solution, 
when applied to the initial event (t,x). I 

The best possible situation is that in which the minimization problem in 
the (HJB) equation has a unique solution as a function of V x and the rest of 
the data, at each (t,x). In order to consider this situation, we introduce the 
Hamiltonian associated to the optimal control problem, 



H(t, x, u, A) := q(t, x, u) + X' f(t, x, u) 



seen as a function of (t, x,u) G [er, r] x X x U and A G R™. Thus, the above 
situation arises when 

min7i(i,a;,u, A) 

u 

exists and is achieved at a unique u for each fixed (t, x, A). Actually, we will be 
interested in the case where this unique minimum is continuous as a function of 
(t,X, A), that is, when there is a continuous function 

a : [er, r] x X x R n -> It (8.23) 

satisfying 

min7i,(t, x, u, A) = Ti.(t, x, a(t, x, A), A) , 

u 

and this is the unique minimum: 

q(t, x, a(t, x, A)) + \'f(t, x, a(t, x, A)) < q(t, x, v) + X'f{t 7 x, v) (8.24) 

for all 

(t, x, A) e[a,T}xlx R n 

and all 

v € IX so that v 7^ a(t, x, A) . 

But even then, this solution when substituted into the system may not result 
in an admissible control. We need for that one more definition: 

Definition 8.1.9 Let £ be a system and letX be an interval in T. A function 

fxi . -L X .A* ' IX 

is an admissible feedback law on X if for each ft€l and each x G X there 
exists a unique trajectory (1;,<-<j) on J = {t\ t G X, t > /J,} such that 

£(/z) = x 

and 

u J (t) = k(t i at)) 
for almost all t G J. In that case u) is called the closed-loop control starting 
at (/i, x). □ 
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As a trivial example, kit, x) := x is admissible on [0, oo ) for the linear system 
x = u since there exist solutions of x = x defined for all initial states and times, 
but the feedback k{t,x) := x 1 is not admissible for this system on the interval 
[0, 2], since solutions do not exist on that interval for x = x 1 and x(0) = 1. 

We summarize next the conclusions of the infinitesimal version of the veri- 
fication result, Lemma 8.1.6, together with the converse provided by the above 
Proposition: 

Theorem 36 Let (OCD) be optimal control data, and assume that 

• there is some continuous function a as in (8.23) satisfying (8.24); 

• there is given a continuously differentiable function V on [a, r] x X so that 

k(t, x) := a(t, x, V' x (t, x)) 
is an admissible feedback on [a, r] . 
Then, the following two statements are equivalent: 

1. V satisfies the HJB equation on (cr, r) x X, which can now be written as 

q(t,x,k(t,x)) + V x (t,x)f(t,x,k(t,x))+V t (t,x) = 0, 
V(t,x) =p{x). 

2. There is a smooth solution of the optimization problem, V is the Bellman 
function, and for each (/j,, x) G [cr, r] x X the closed-loop control starting 
at (fJ-,x) is the unique optimal control for (/i,a;). 

Proof. That (2) implies (1) follows from Proposition 8.1.8. We now prove the 
converse, using the verification Lemma 8.1.6. 

Pick any a < t < s < r and any x, and let v be the closed-loop control 
starting at (t,x) up to time s. Let ui be any control on [t, s] that is admissible 
for x. Denote £ := tp(x,uj) and consider the function -B(r/) := V(t],S,(ji)) on 
[t,s]. This is absolutely continuous, and its derivative is 

Vtfativ)) + V x (ri,$(r,))f(Ti,$(ri)Mv)), 

which, since V satisfies the HJB equation, is the same as 

This last expression is, because of property (8.24), greater than or equal to 

and equal to this when to = v. So 

V{s, £(*)) - V(t, x) > - J q(n, ${v),w{v)) dv 
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with equality when w = v. By continuity on t and s, the same inequality holds 
even if t = a or s = t. So the hypotheses in Lemma 8.1.6 hold. We conclude 
that v is indeed optimal for (fi,x), and it is continuous because a is. 

Finally we prove the uniqueness statement. Assume that lu differs from v on 
a set of positive measure. Then dB/drj is strictly greater than -?(»/,((l),w()/)) 
on this set, and hence it follows that 

V{t,x) < J(r,t,x,w) 
and thus lu cannot be optimal. I 

Remark 8.1.10 A particular situation in which a function a such as required 
in the hypothesis of Theorem 36 will exist is as follows. Assume that the system 
£ is affine in the control, meaning that 

f(t, x, u) = A(t, x) + B(t, x)u , 

where A is an n- vector, and B is an n x m matrix, of C 1 functions of (t,x). 
This happens for instance if we are dealing with continuous-time linear systems 
(with A(t),B(t) continuously differentiable) , in which case A(t, x) = A(t)x and 
B(t,x) = B(t). 

Assume also that the instantaneous cost is quadratic in u, meaning that we 
can write 

q(t, x, u) = u R(t, x)u + Q(t, x) , 

where R and Q are continuously differentiable functions of (t,x), Q is nonneg- 
ative for all (t,x), and R is an m x m symmetric positive definite matrix for 
all (t,x). We then may "complete the square" and write (omitting arguments 
(t,x)) 

u'Ru+X'Bu = (u+-R- l B'\\ r(u+-R- 1 B'\\ - jX'BR^B'X 

for each (t, x,X) G [a, r] x X x K". It follows that the unique minimum of the 
Hamiltonian 

q(t, x, u) + X'f(t, x, u) 

with respect to u is given by 

a(t,x,X) := --R^B'X. 

Note that this defines a continuous (in fact, differentiable) function a. □ 

A special case of the situation discussed in Remark 8.1.10 is that of con- 
tinuous-time linear systems x = A(t)x + B{i)u over K = R, when we want to 
minimize a quadratic cost with q(t, x, u) := v! R(t)u+x' Q(t)x, and p{x) := x'Sx. 
(Here R, Q, and S are matrix functions, S constant, of appropriate sizes.) We 
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should expect the Bellman function, if it exists, to be quadratic in x. (Intu- 
itively, we argue as follows. For given initial (jx,x), and any given control lu, we 
get the state evolution £(£) = $(£, n)x + / $(£, s)B(s)co(s)ds, and when substi- 
tuted back into the cost J(fj,,T,x,u>), this cost becomes quadratic on lu and on 
the initial state x. As a function of lu, we then expect that there will be a unique 
minimum for each x, and that this minimum will be achieved at an u> which de- 
pends linearly on x. When this minimum is substituted in the cost function, we 
get finally a (homogeneous) quadratic function of x.) Thus we expect to be able 
to write V(x) = x'Px, for some symmetric matrix P, from which it follows that 
the gradient of V is linear on x: V x (x) = 2x' P . But then, the feedback law in 
Theorem 36 (p. 361) becomes k{t,x) = a(t,x,Vj.(t,x)) = -R^^B^t)' P{t)x , 
which is linear in x. Substituting in the dynamics of E, we get the closed-loop 
equations (omitting all t arguments): x = (A + BF)x, F = —R~ l B'P. This 
is again a linear system, so it is complete. Thus k is an admissible feedback 
law. In summary, if V is quadratic, then the optimal controls are induced by 
linear feedback. Moreover, it turns out, the HJB equation for V(t,x) = x'P(t)x 
is equivalent to an ordinary (not partial) differential equation on the matrix 
function P, the Riccati Differential Equation. 

In order to allow the study of the linear case by itself, we next discuss it in a 
self-contained manner; however, the previous discussion explains the motivation 
for the steps to be taken. We return to the general continuous-time nonlinear 
case in Section 8.5. 

8.2 Linear Systems with Quadratic Cost 

We consider continuous-time linear systems (with K = R): 

x{t) = A{t)x{t) + B{t)u{t) . (8.25) 

We assume that A and B are measurable essentially bounded (m.e.b.). Also 
given are three matrices of functions R, Q, S such that: 

• R is an m x m symmetric matrix of m.e.b. functions, Q is an n x n 
symmetric matrix of m.e.b. functions, and S is a constant symmetric 
n x n matrix; 

• For each t, R is positive definite; 

• For each t, Q is positive semidefinite; and 

• S is positive semidefinite. 

For each pair of real a < r numbers, each state x G R", and each measurable 
essentially bounded input lu : [o~,t] — * R m , we define: 

J(t,*,x,w) := [ w(8)'R(a)w(a)+ ^(8)'Q(a)^8)da + S(t)'SS(t), 
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where £(£) = (f>(t,o~,x,u>) for all t £ [a, r]. The problem that we wish to study 
is this one: 

For each pair of times a < r and state x°, find a control lu which 
minimizes J(t, cr, x°,lu). 

Any control u> with this property will be said to be optimal for x° on the interval 
[a,r]. 

Assume a < r have been fixed. We denote 

V{t, x) := inf {J(t, t,x,u)\u: [a, r] -> R m m.e.b. } (8.26) 

for each t £ [a, r] and x £ R™. This is called the value or Bellman func- 
tion associated to the optimization problem. If a control cu° which minimizes 
J(t, t, x°, u) exists, then V(t, x Q ) = J{t, t, x° , u>°). 

Let F : [a,r] —> E mx " be m.e.b.. For each t° £ [a,r] and x° £ K™, consider 
the solution x(t) of 

±{t) = [A(t) + B{t)F{t)]x{t) , x{t°) = x° 

which is obtained when using the feedback law u(t) = F(t)x(t) with the sys- 
tem (8.25), starting from state x Q at time t . The control ut(t) = F(t)x(t) is 
called the F -closed-loop control starting at (t a ,x°). 
The main result will be as follows. 

Theorem 37 Pick any a < r. Then there exists an absolutely continuous 
symmetric nx n matrix function P{t), defined for t £ [a, t], which satisfies the 
matrix Riccati Differential Equation on [o~, r]: 



P=PBR- X B'P- PA- A'P-Q, P(t) = S (RDE) 



Let 



F(t) := -R{t)- 1 B{f)'P{t) . (8.27) 



Then the F -closed-loop control starting at (<r,x°) is the unique optimal control 
for x a on the interval [<j,t]. Moreover, 

V{t,x) = x'P{t)x 

for all t £ [cr, r] and x £ R n . 

The main technical step is to establish the conclusions under the assumption 
that P exists: 

Lemma 8.2.1 Pick any a < t. Suppose that there exists a matrix function 
P(t) as in the statement of Theorem 37. Then, with F(t) = -i?(t)" 1 B(t)'P(t), 
the _F-closed-loop control starting at (a, x ) is the unique optimal control for x° 
on the interval [cr, t], and V(t, x) = x 1 P{t)x for all t £ [a, r] and x £ R n . 
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We will provide two independent proofs of this Lemma. The first proof 
is short and uses a simple algebraic computation. The second proof is much 
longer, but it has the merit of being conceptual and to generalize to large classes 
of nonlinear systems (it is a particular case of the approach discussed in Sec- 
tion 8.1). Before providing these proofs, we show how the Theorem follows from 
the Lemma. 

We must show for that purpose that the final-value problem (RDE) has a 
solution on all of the interval [a, r] . Then it will follow that the resulting P must 
be symmetric, since its transpose P(t)' solves the same final value problem and 
solutions of (locally Lipschitz) ODE's are unique. 

Consider the final-value problem P = PBR^B' P-PA-A' P-Q, P(t) = S, 
on the interval X = [a, r]. Let P(-) be its maximal solution, defined on some 
interval JCI which is open relative to X. We wish to prove that J = X. Pick 
any a' € J. Since there does exist a solution of (RDE) on the interval [ct',t], 
we may apply Lemma 8.2.1 to the optimal control problem on the restricted 
interval [it', t], and conclude that V(t, x) = x'P(t)x is the Bellman function for 
the corresponding problem on that interval. Pick any x £ 1™ and any t € [a' ,t]. 
Then, 

0<V(t,x) < J(r,t,x,0) 

= x'®(T,t)' \ f ®(s,T)'Q(s)${s,T)ds + s\®(T,t)x. 



In particular, this applies when t = a' . As functions of t € [a, r], both 3>(t, i) 
and the expression in braces {. . .} are bounded. Thus, there is some constant 
K > such that 

< x'P{a')x < K\\x\\ 2 

for all x G 1™. Since the norm of the symmetric and nonnegative matrix P(cr') 
equals the largest possible value of x'P(a / )x, \\x\\ = 1, we conclude that 

\\P(*')\\<K. 

Therefore, the maximal solution P(-) stays in some compact subset of the set 
of all real n x n matrices. By Proposition C.3.6 (applied to the time-reversed 
equation), it follows that P is defined everywhere, that is, J = X, as we wanted 
to show. This completes the proof of Theorem 37, under the assumption that 
Lemma 8.2.1 holds. I 

First Proof of Lemma 8.2.1 

Suppose that P satisfies (RDE). Pick any initial state x° and any control 
u) : [a, t] —* W 71 , let £(£) = cj)(t, a, x , w), and consider the time derivative 
of the function £(t)' P(t)£(t) (defined almost everywhere). Using (RDE) and 
£ = At; + Buj, we have: 

(£'P£)' = {uJ + FT l B'Pti)'R{uj + R- l B'Pti)-uj'Ruj-£'Qt. 
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Integrating, we conclude: 

J(t,<t,x°,u) = x 'P{a)x° + 

(u)(s) + i?- 1 (s)B(.s)'P( S )e(s))'i?(a;(s) + i?- 1 (s)B(.s)'P(s)e(s)) ds . 

From this equality, it is clear that the unique minimum-cost control is that 
one making u>(s) + R~ 1 (s)B(s) / P(s)£(s) = 0, and that the minimum cost is 
X°'P{a)x°. I 

Second Proof of Lemma 8.2.1 

Assume that P satisfies (RDE). Denote, for each t,x, V(t, x) := x'P(t)x. 
(Of course, we do not know yet that V is indeed the value function, but the 
notation should not cause confusion.) We also write: 

q(t, x, u) := u'R(t)u + x'Q(t)x , p(x) := x'Sx , 

and 

k(t, x) := F{t)x , f(t, x, u) := A{t)x + B(t)u . 

Since V x (t,x) = 2x'P{t) and 

x'[PBR- 1 B'P - 2PA - Q]x = x' [PBR^B'P -PA- A'P -Q]x, 

the property that P satisfies (RDE) translates into the fact that V satisfies the 
following Hamilton-Jacobi-Bellman partial differential equation: 

V t (t, x) + q{t, x, k(t, xj) + V x {t, x)f{t, x, k{t, x))=0, V(t, x) = p(x) . 

We need next this observation: 
Claim: Let R be any m x m symmetric positive definite matrix, B € M nxm , 
and A € R™. Then the minimum value of v! Ru + X' Bu over all u G R m is 
achieved at the unique point u = — ^R -1 B' \ . 

Proof of claim: We just "complete the square" : 

u'Ru+X'Bu = (u+-R- l B'\\ r(u+-R- 1 B'\\ - -X'BR^B'X. 

In particular, when A' = V x (t,x) = 2x'P(t), this implies that 

u = --7?(t)" 1 B(t)'A = F(t)x = k{t,x) 

minimizes q(t, x, u)+V x (t, x)f(t, x, u). Thus the Hamilton-Jacobi-Bellman equa- 
tion translates into: 

V t (t,x) + min {q(t,x,u) + V x (t,x)f(t,x,u)} = (8.28) 

with the boundary condition V(r, x) = p(x). (This is the form of the usual 
HJB equation associated to the minimization of a cost criterion J q(t, x, u)dt + 
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p{x(r)) for a general system x = f{t,x,u), cf. Section 8.1. For linear systems 
with quadratic cost, saying that a quadratic-in-x function V = x'Px solves 
the HJB equation is exactly the same as saying that P satisfies the RDE.) Yet 
another way of stating these facts is by means of this pair of equations: 

V t (t,x) + V x (t,x)f(t,x,k(t,x)) = -q(t,x,k(t,x)) (8.29) 

V t (t,x) + V x (t,x)f(t,x,u) > -q(t,x,u) yu^k{t,x), (8.30) 

understood as being true for almost all t G [a, r] (namely those t for which P(t) 
exists) and all x € E". 

Pick any x° G K". We must prove that the _F-closed-loop control starting 
at (a, x Q ) is optimal. Call this control u; and let £ be the corresponding state 
trajectory. Let a(t) := V(£, £(£)), and observe that 

<*(t)=p(S(t)) (8.31) 

because V satisfies the boundary condition V(r, x) = p(x). Equation (8.29) 
implies that a(t) = —q(t,£(t),u>(t)), from which, integrating and using Equa- 
tion (8.31) we conclude that 

P (1(t)) = a(a) - f q(t,l(t)Mt))dt=V(a,x )- f q(t,^(t),w(t))dt 

and therefore V(a, x°) = J(t, a, x , w). 

To show that V is the value function and that w is the unique optimal control 
for x° on the interval [a, r], we must verify that V(a, x Q ) < J{t, a, x° , lu) for all 
other controls u>. So pick any other u : [<j,t] — > K m , and let ^(i) = </>(£, a, x , u>) 
be the solution of £(t) = f(t,£(t),u)(t)) with £,(&) = x°. If it were the case that 
u>(t) = k(t, ^(t)) for almost all t G [<j, r], then ^ would satisfy ^ = f(t, £, k{t, ^)), 
so by uniqueness of solutions £,=£,, and thus uj = TD, which is a contradiction. 
Therefore we know that u>(t) ^ k(t,t;(t)) for all t in some subset / C [a, r] of 
positive Lebesgue measure. Write /?(£) := V(t,£(t)). We have that 

0(t)= P (S(t)) (8.32) 

because V satisfies the boundary condition V(r, x) = p(x). Equation (8.30) 
implies that (5(t) > -q{t,£{t),uj{t)) for all t e I and /3{t) = -q(t,t(t),w(t)) for 
all other t, so integrating and using Equation (8.32) we conclude that 

P (S(t)) > P(a) - [ q(t,at)^(t))dt=V(a,x )- [ g(t,^(t),w(t))dt 



and therefore V(a,x°) < J{t, a, x ,uS), as desired. I 

The linear quadratic problem solved above is often called the standard reg- 
ulator problem. The solution of the Riccati equation is computed numerically. 
Note that, by symmetry, one must integrate a system of n(n+l)/2 simultaneous 
nonlinear differential equations. Typically, only the limiting solution, described 
later, is computed. 
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Remark 8.2.2 Note that the matrices P(t), though symmetric, are not neces- 
sarily positive definite. For instance, if 

S = and Q = , 

then J is always minimized by u> = 0, and the optimal cost is therefore 

V(t,x) = 

so that P = 0. Equivalently, the matrix P = solves the Riccati equation 
(RDE) when S = Q = 0. □ 

Exercise 8.2.3 Consider the regulator problem for the system (with n = m = 
1) on [0, 1], x = u, with Q = R = 1, S = 0. Find explicitly the form of the 
optimal feedback kit, x). □ 

Exercise 8.2.4 {Optimal servo problem.) Many problems of interest can be 
reduced to the standard regulator problem. One of these is the servo problem 
defined as follows. The objective is to have the output y(t) = C(t)x(t) of 
a linear system with outputs £ closely follow a desired reference signal r(t). 
Mathematically, we wish to minimize, for given x and r(-), a cost J (a, r, x, lu) 
which here equals 

{u>(t)'R(t)u(t) + (C(t)t(t) - r(t))' Q(t) (C(t)Z(t) - r(t))} ds 

+ (C(rX(r)-r(T)) / 5(C7(rX(r)-r(T)), 

where (£, ui) is the trajectory with £(cr) = x. In one version of this problem, one 
assumes that r(t) is produced by a linear control generator, that is, r is itself 
the output of a continuous-time linear system with outputs (and no controls), 
say 

z(t) = A z{t) 
r(t) = C Q z(t). 

Assuming that R(t) is positive definite and Q(t) is positive semidefmite for all 
t, show how to find a solution for this problem, and show that the solution is 
unique. (Hint: Introduce a large linear system, whose state space consists of 
all pairs ( x ) . Apply the regulator results to this extended system. The optimal 
feedback will be a function of t,x, and z.) □ 

Remark 8.2.5 It may appear that restricting to signals which are outputs of 
linear systems is not very interesting. On the contrary, many signals of interest 
can be so represented. For instance, any polynomial of degree d can be obtained 
as the output (for a suitable initial state) of a system with y( d+1 ) = 0, for 
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instance the output of the system of dimension d + 1 and 
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C=(1,0,...,0) 



In practice, z(i) itself is not directly available to the controller, so it must be 
estimated in order to apply the solution obtained in the above problem. For 
instance, in the case of polynomial signals as in this example, estimation of z(t) 
could be done by derivation of the reference signal r(t), or by algorithms using 
observer theory (see later) that do not involve differentiation. □ 

Exercise 8.2.6 It is sometimes of interest to consider costs that penalize cor- 
relations between input and state variables, that is, to include a cross-term of 
the type uj(i)' L(i)^(t) in q. (For instance, it may be physically undesirable to 
apply a negative u\ when X2 is positive, in which case we can add a term such 
as — pLOi{t)^2{i) for some large p > 0.) Specifically, assume that 



q(t, x,u) = (u 1 x') 



R L 
L' Q 



where R(-) is positive definite and where the composite matrix (possibly time- 
varying) 

' R L 
V Q, 

is positive semidefinite. Show that Theorem 37 (p. 364) remains valid in this 
case, with a slightly modified feedback law and Riccati equation. (Hint: Reduce 
to the case L = by completing the square.) □ 

Exercise 8.2.7 Obtain an analogue of Theorem 37 (p. 364) for linear discrete- 
time systems with costs of the type 

r-1 

J2Mk)'R(kMk) + akYQikm)} + ^(r)'S^(r) . 

k— a 

(The Riccati equation is now of course a difference equation; its right-hand side 
is a bit more complicated than before.) □ 



Hamiltonian Formulation 

In general, a Hamiltonian matrix (of size 2n) is a 2n x 2n real matrix H that 
satisfies 

H'J +JH = 0, 
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where 

and / is the n-dimensional identity matrix. Equivalently, H has the form 

' M L 



H ~ ' N -M' 

for some symmetric N and L and arbitrary M. 

In particular, we associate to the matrix functions A,B,R,Q the matrix 
function 

A -BR~ 1 B r 



H ' -Q -A' 

which is a Hamiltonian-matrix valued function H(t),t € [a, r]. The following 
fact is proved by a straightforward computation: 

Lemma 8.2.8 Let £,oj be a pair consisting of an optimal trajectory start- 
ing from a state £(cr) = x°, and the corresponding optimal control u)(t) = 
-R(t)- 1 B(t)'P(t)£(t) on the interval [a,r\. Then: 



I(a) =h (a)' ZW = x °' a W = ^W ( 8 - 34 ) 

where A(t) := P(t)£(t). D 



Thus, along optimal trajectories, the vector (£(£)', A(i)')' solves the two-point 
boundary value problem (8.34). This is the form of the "maximum principle" 
statement of optimality; we show next that in this way one obtains a unique 
characterization of the optimal trajectory: 

Lemma 8.2.9 For each A, B, Q, R, S (m.e.b. entries), there is a unique solution 
of the two-point boundary value problem (8.34). 

Proof. Existence has been proved (use the optimal trajectory and A = -P£). We 
must show uniqueness. For that, it will be sufficient to show that every solution 
of the two-point boundary value problem (8.34) *- s a solution of the following 
initial value problem: 



p, = Hn, fi{a) = I J . (8.35) 

Let P(-) be the solution of (RDE), and consider the unique solution on the 
interval [a, r] of the following matrix linear differential equation: 

X= (A-BR- 1 B'P)X, X(t) = I. (8.36) 



8.3. Tracking and Kalman Filtering* 371 

We let A(i) := P{t)X{t) and 

* (i) := (a(*)) ' sothat * (r)= (s 

A simple calculation using that P satisfies (RDE) and X satisfies (8.36) shows 
that this In by n matrix function solves the corresponding matrix Hamiltonian 
differential equation: 

* = HV . 

Now pick any solution u(t) = (£(£)', X{t)')' of the two-point boundary value 
problem (8.34). Observe that u(t) is in the kernel of the n by In matrix [S , —I] 
and that the columns of \&(t) are n linearly independent vectors in this (n 
dimensional) kernel. Thus, there is some vector q G R™ so that u(t) = 9(r)q. 
Since p = u — ^q satisfies p = Hp, p(r) = 0, it follows that /z = *&q. Thus, in 
particular, £(<r) = X(<r)q, and also 

X(a) = A(a)q = P(a)X(a)q = P(v)Z(<r) , 

which is the claim that had to be proved. I 

Remark 8.2.10 Observe that the first n rows of ^(i), that is, X(t), form a 
nonsingular matrix for each t, since X(t) is a fundamental matrix associated 
to x = {A - BR- 1 B / P)x. Since, by definition, A(t) = P(i)X(i) for all t, we 
conclude that 

P(t) = AftX®- 1 

for all t. This provides a way to solve the nonlinear differential equation (RDE) : 
first solve the final value problem VP = H^S , ^(r) = [I,S']', partition as ^ = 

[X 1 , A']', and then take P = KX' 1 . ' U 

8.3 Tracking and Kalman Filtering* 

We now study a problem of output tracking, and, as an application, a deter- 
ministic version of Kalman filtering. 

Tracking Under Disturbances 

A problem which arises often is that of finding controls so as to force the output 
of a given system to track (follow) a desired reference signal r(-). We also allow 
a disturbance ip to act on the system. 

Let E be a linear system (over K = R) with outputs 

x = A(t)x + B(t)u + ip(t), y = C(t)x, 



* This section can be skipped with no loss of continuity. 
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where tp is an K n -valued fixed m.e.b. function. Assume given an 
fixed m.e.b. function r. We consider a cost criterion: 



-valued 



J(T,a,X°,0J,cp, r) :-- 



where £ = A£ + Bu, £(ct) 



w(t)'R(t)w(t) + e(t)'Q(t)e(t)dt + f(r)'S£(r) 



and e := C£ — r is the tracking error. As in the 



quadratic optimal control problem, it is assumed that: 

• R is an mxm symmetric matrix of m.e.b. functions, Q is a pxp symmetric 
matrix of m.e.b. functions, and S is a constant symmetric n x n matrix; 

• for each t, R is positive definite; 

• for each t, Q is positive semidefinite; and 

• S is positive semidefinite. 

The problem is, then, that of minimizing J , for a given disturbance ip, reference 
signal r, and initial state, by an appropriate choice of controls. Observe that 
in the special case when ip = r = this is a standard linear-quadratic problem, 
with "Q" matrix equal to C'QC (cost imposed on outputs). For p = and r 
arbitrary, we have a tracking problem; for r = and <p arbitrary a problem of 
regulation under disturbances. 



Figure 8.1: Tracking problem. 

This problem of tracking under disturbances can be reduced to an optimal 
control problem for a linear system with quadratic costs, as follows. The idea 
is very simple: add a constant state x n+ i = 1, allowing the incorporation of ip 
into A and of r into Q. We do that next. 

On the interval [a, r] we consider the system (with no outputs) 



where 



Ax + Bu, 



and B 



have sizes (n + 1) x (n + 1) and (n + 1) x m, respectively. We also introduce 
the associated cost 



J(r,CT,:r ,^,u;) : = 



w(t)'R(t)w(t)+$(tyQ(t)Z(t)dt + Z(t)'SZ(t), 
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where tildes on x°, £ are used to indicate vectors and vector functions of dimen- 
sion n + 1 . The new matrices are defined as 

C'QC -C'Qr\ s _ (S 
-r'QC r'Qr ) ^00 

(from now on, we drop arguments t whenever clear from the context). In terms 
of these data, 

J(t, a, x°,U)) = J(r,a,x°,u),ip,r) 

for each x°,u>, where x° := (a; , 1)'. Note that the resulting problem will have 
time-dependent costs even if the original system was time-invariant and the 
matrices Q,R were constant. The matrices Q and S are still semidefinite; in 
the case of Q this is a consequence of the formula 

(V a) Q(t) ( Z \={Cz- ar{t))'Q{Cz - ar{t)) , 

which holds for each z £ W 1 and each oGR. 

Minimizing J with initial states of the form x° = (x Q , 1)', for the system 
(A,B), provides a solution to the tracking problem. This is achieved using the 
results earlier in the Chapter. Let P be the solution of the Riccati Differential 
Equation (RDE) on page 364, 



P=PBFT 1 B'P-PA-A'P-Q, P(t) = S , 

which one knows exists on all of [a, r], by Theorem 37 (p. 364). This solution 
can be partitioned as 

P ff 
J3> a, 

(all functions of t), where P is n x n and a is scalar. In terms of these data, the 
(RDE) takes the form of the set of equations 

P=PBR- 1 B'P-PA-A'P-C'QC, P{t) = S (8.37) 

which is itself the Riccati equation for the optimization problem that results 
when ip = r = and can be computed independently of the particular r and tp 
of interest, together with 

P=-(A + BF)'p+C"Qr-Ptp, /3(r) = (8.38) 

and 

a = p'BR- 1 B'f3-r'Qr-2tp'f3, a(r) = 0, (8.39) 

where we denoted 

F := -R-i-B'P 
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just as in the standard optimal control problem. For an initial state of the form 
x° = (a; , 1)' the optimal path satisfies x = (A + BF)x, which gives 

x = (A + BF)x-BR- 1 B'f3, x{a) = x° , (8.40) 

and the optimal control u = — Rr x B'Px can be written in the form 

u = Fx-R- l B'[3 (8.41) 

while the optimal cost is 

(x^yPiaYx 5 = (x yP(ayx a + 2(x a yp(a) + a(<7) 

(this is the only place where a appears). Note the form of the solution: The 
tracking signal and the disturbance drive the system (8.38), which is a system 
that is formally adjoint to the closed-loop system obtained for no tracking, and 
the state (3 of this adjoint system in turn appears as an extra term in the closed- 
loop system. When r = and tp = 0, also a = (3 = 0, and the solution to the 
standard regulator problem is recovered. 

The discussion can be summarized as follows. 

Theorem 38 Assume that A, B, C, Q, R, S are matrix functions as above, and 
let a < t. Then the solution of the final-value problem, (8.37) exists on [a, r]. 
Given any m.e.b. functions ip and r on this interval, and any x° , solve (8.38) 
and (8.40) to obtain x(-) and f3(-) on [a, r]. Then the control u defined by (8.41) 
has the property that it minimizes 

{u'Ru + (Cx - r)'Q{Cx - r)} dt + x{t)'Sx{t) 

subject to x = Ax+Bu+(p, x(o~) = x° , among all measurable essentially bounded 
controls, and is the unique such control. Moreover, the minimum value of this 
integral is given by 

x P{a)x + 2{x Yf3{a) + a, 

where a = - /J"(/3' BR^B 1 '[3 - r'Qr - 2ip'f3)dt. □ 

Remark 8.3.1 If S is positive definite, then the matrix P(o~) — which depends 
on the data A, B, R, Q, C but not on the particular x° , ip, and r — is also positive 
definite. This can be proved as follows. Assume that 

{x°yp{o-)x° = o 

for some nonzero x , and consider the tracking problem with ip = r = 0. The 
optimal cost is zero, since (3 = a = 0, so the cost along the optimal trajectory 
(£, u>) vanishes: 

I {lu'Rlo + i'C'QCi] dt + ^{t)'S^{t) = , 

J a 



8.3. Tracking and Kalman Filtering* 375 

which implies u) = and £(t) = by positive definiteness of R and S. In 
conclusion, one has a solution of 

Z = M 

for which £(r) = 0, which implies £ = 0, so a; = £(0) = 0, as wanted. □ 

Exercise 8.3.2 Let a < r and assume that Q is positive definite and {A, C) 
is observable on [a, r] (but /S is merely semidefinite). Show that one can again 
conclude that P(o~) is positive definite. □ 

(Deterministic) Kalman Filtering 

Let (A, B, C) be a time- varying continuous-time linear system as in the previous 
section, and let Q, R, S be m.e.b. matrix functions of sizes n x n, m x m, and 
n x n, respectively. Now we assume that both S and R(-) are positive definite. 
The problem to be considered in this section is as follows: For any given 
m.e.b. function y on an interval [a, r], minimize the expression 

[u(t)'R(tMt) + (C(t)m - y(t))'Q(t)(C(t)m - y(t))}dt + S(v)'S£(<r) 

(8.42) 
over the set of all possible trajectories (£,u>) of £ = ^4£ + Buj on [a, r]. 

Remark 8.3.3 This problem is a minor variation of the tracking problem (with 
no disturbances, that is, with tp = 0) considered in Theorem 38 (p. 374). The 
only differences are in the imposition of an initial as opposed to a final state 
cost and — more importantly — the fact that the minimization is to be carried 
out over all possible trajectories, not merely among those that start at a fixed 
initial state. The motivation for studying this question is as follows. Consider 
the system 

x = Ax + Bu, x(a) = x 
y = Cx + v , 

where the functions u and v are thought of as independent and unknown pertur- 
bations affecting the system, and the initial state x° is unknown. The function 
y represents an observation of y on the interval [a, r], and it is desired to obtain 
an estimate of the state trajectory, and in particular of the final state x(t), 
based on this observation. Moreover, one wishes to obtain the "best possible" 
estimate, where "best" is taken to mean the one for which the disturbances u 
and v, as well as the initial state x , would have been as small as possible. The 
matrices S, Q, and R are used to quantify the relative weighting to be given 
to each component of the uncertainty. For instance, if it is known that the 
measurement noise v on some component of the output is very large compared 
to other components and to the uncertainty in the initial state and the state 
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noise, then one should tend to assign less value to these observations, that is, 
the corresponding entries of Q should be taken small. 

The same question can be interpreted in statistical terms, and then it is more 
properly called the Kalman filtering problem. In that version, u(-) and «(•) are 
independent white noise processes and x° is a random vector, all assumed to be 
Gaussian distributed and zero mean. With R,Q, and S being, respectively, the 
inverses of the covariance matrices of u,v, and a; , the minimization problem 
that we posed is equivalent to that of finding the minimum variance estimate 
of x given the observations y (a conditional mean). The cost function (8.42) is 
the likelihood function of the problem. The terminology "filtering" reflects the 
goal of filtering-out the noise in order to recover the state cc(-), interpreted as 
the signal of interest, from the measured signal y. (Actually, we solve what is 
called the "smoothing" problem first.) □ 

Reduction to Tracking Problem 

The filtering problem is reduced to the tracking problem (with <p = 0) by first 
reversing time, so that the cost is imposed on the final state, and then minimiz- 
ing over all possible solutions to the tracking problem, over all possible initial 
states. Finally, and most importantly, a recursive form of the solution is derived, 
which allows updates of the state estimate as new measurements are taken. Let 

R(t) := R(r + a-t), Q(t) := Q{t + a - 1) , 
C{t) := C(T + a-t), r{t) := y (r + a - 1) 

and 

A(t) := -A(r + a-t), B{t) := -B(r + a-t) 

for all t € [a, r], for any given y. For each trajectory (£,w) of (A,B), 

f(t) := Z(T + (J-t), u(t) := ui(T + a-t) 

provide a trajectory of (A,B) and conversely. 
In terms of these data, one can write (8.42) as 

{u(t)'R(t)w(t) + (C(t)£(t) - r(t))'Q(t) (C(t)t(t) - r(t)) } dt + ^'^(r) , 

(8.43) 
and the problem becomes that of minimizing (8.43) over all trajectories of 
(A, B). For each fixed x° this minimum is provided by Theorem 38 (p. 374). 
We use tildes for the solution of the corresponding Riccati equation: 

P=PBRT 1 B'P-PA-A'P-C'QC, P(t) = S . (8.44) 

The other relevant equations are, then: 

/?= PBR^B'fi-A'P+C'Qr, J3(t) = (8.45) 
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and, with 

F = -Rr x B'P, 

then 

x = (A+ BF)x- BR- X B'[}, x(a) = x° (8.46) 

is the optimal path. Also from the Theorem it follows that the cost along this 
trajectory is 

{^)'P{a)x^ + 2(sc°) / 0(<r) + a, (8.47) 

where a is a constant that does not depend on x°. Minimizing the expression 
for x° will provide the minimum for the desired problem. By Remark 8.3.1, 
-P(cr) is invertible, so (8.47) equals 

p + P(a)" 1 /3(a))'p(a)p + P(a)- 1 /3( ( r)) - /^yP^)" 1 /?^) + a, 

which admits the unique minimum 

£5 = -p^y^a) . (8.48) 

This gives a solution to the observation problem: First solve for P and (3, then 
for x using the initial condition (8.48), and finally, reversing time, obtain P, /3, x. 
The details are as follows. 
Consider the equations: 

P = -PBR- 1 B'P- PA- A'P+C'QC . P(a) = S (8.49) 

(3 = -PBR- l B'(3 - A'f3 - C'Qy , /3{a) = (8.50) 



as well as 
and 



F = -R^B'P 



{A- BF)x + BR~ 1 B'(3, x(t) = -P(r)" 1 (3{t) . (8.51) 



Theorem 39 Assume that (A,B,C) is a continuous-time linear system with 
outputs, R and S are positive definite as above, and all entries of A, B,C, R,Q 
are m.e.b.. Then, the solutions of (8.49) and of (8.50) exist on each interval 
[a,r]. 

Given any m.e.b. function y on any interval [a, t], let £(•) be the solution 
of (8.51) on this interval. Then, there exists some lu(-) so that (£,^->) is the 
unique trajectory minimizing (8.42). □ 

Observe that the final state x(t) for the optimal solution can be obtained 
by solving the differential equations for P and (3 and using simply x(t) = 
— _P(t)~ 1 /3(t). Thus this value x(t), the "filtered" estimate of the state, can be 
computed "online" on increasing time intervals (increasing t). One can even 
avoid the matrix inversion, resulting in a recursive estimate, as follows. 
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Recursive Formulas for State Estimate 

With the previous notations, we let 

U(t):=p-\t) 

for all t G [a, r] . Consider the vector 

z(t) = -Tl(t)(3(t) 

for t G [a, t\. It follows from the above discussion that z(t) is the same as the 
final state x(t) for the optimization problem restricted to [0, i\. 

From the fact that II = — IIPII, substituting P from (8.49), and using that 
HP = I, we conclude that II satisfies the Filtering Riccati Differential Equation, 
or "dual RDE" , 



ri = iL4' + aii - nc'Qcn + br~ x b' , n(o-) = s- 1 . (frde) 



Furthermore, from (FRDE) and (8.50) we conclude that 

z = (A - IlC'QC)z + IlC'Qy, z(a)=0. 

With the notation 

L := -TIC'Q, 

this can also be written as 

z = Az + L[Cz-y], z{a)=0 (8.52) 

which is the final form of the equation for z. We can sumarize as follows: 

Theorem 40 Assume that (A, B, C) is a continuous-time linear system with 
outputs, R and S are positive definite as above, and all entries of A, B, C, R, Q 
are m.e.b.. Then, the solution of (FRDE) exists on each interval [a, r]. 

Given any m.e.b. function y on any interval [a, r], let £(•) be the solution 
of (8.52) on this interval. Let (£,w) be the unique trajectory minimizing (8.42). 
Then, Q{t) = £(r). □ 

Remark 8.3.4 Equation (8.52) has the form of the observer derived earlier, in 
Chapter 7. (The control does not appear, but see Exercise 8.3.9 below.) Thus, 
the Kalman filter is an observer where the matrix L has been chosen using an 
optimization criterion (see Remark 7.1.7). Note also that (FRDE) can be solved 
independently of the particular observations; in applications one often precom- 
putes the form of II. Alternatively, one may solve the differential equation online 
— since (FRDE) is an initial-value rather than a final-value problem, it is pos- 
sible to solve for II (r), for increasing values of t, as observations are received. 
Similarly, the estimate z(t) can be computed recursively for increasing values 
of r, using (8.52), and at each r it provides the best estimate of x(r) based on 
the observations {y(t),t s [a, r]}. □ 
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Exercise 8.3.5 Derive explicit formulas for the control lu and the correspond- 
ing optimal cost in the filtering Theorem. □ 

Example 8.3.6 Consider the problem of estimating an unknown constant x° 
subject to additive noise, that is, estimating 



y = X° 



from 



We take the system 



y(t), t>0. 



o, y 



and choose S = 1, Q = q > 0, and R arbitrary (since B = 0, we may take 
m = 1 and R = 1; all formulas including R have B as a factor, so they vanish 
anyhow). A large q should be used if v is expected to be small compared to 
the uncertainty in x°, and vice versa (recall Remark 8.3.3). There results the 
equation 

ri = -n 2 g, II(0) = 1, 

so 

n(t) 



i 



l + qt 



and from here the estimator 



*(*) = TTlTq {y{t) ~ Z{t)) ' 

Note that for q large (small noise) and t small, the solutions approach y very 
quickly, consistent with the intuition that the measurements are very reliable. 
On the other hand, for q small (large noise), z changes slowly, again as expected: 
More measurements are needed before an accurate estimate is obtained. □ 

Exercise 8.3.7 Compute explicitly the optimal filter for the case of 

x = u , y = x + v 

and 

R= S=l, Q = q>0. 

Analyze the behavior of the filter for q large or small. □ 

Exercise 8.3.8 Consider the case when B = 0, Q = I, and S approaches 
zero. Show that the formulas for least-squares observation in Section 6.3 can 
be recovered from the results in this section. (Hint: The equation for P can 
be solved with final condition zero, and its solution at the initial time can be 
expressed directly in terms of the Wronskian W.) □ 
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Exercise 8.3.9 Consider the problem of minimizing the same cost (8.42) over 
the set of all trajectories of the system 

£ = AZ + Gu+Blu, 

where u is a known control on [a, r] . Show that the minimum is achieved through 
the solution of 

z = Az + L[Cz -y] + Gu, z(a) = , 

where L is the same as before. (Hint: Simply convert the original problem 

into an estimation problem for £ = A£ + Blu, where £ := £ — x and x satisfies 
W=Ax + Gu.) □ 

8.4 Infinite-Time (Steady-State) Problem 

We next consider the case of optimal controls for costs functions as earlier, but 
now for the problem on an infinite interval. To simplify and because this is the 
most interesting case when r = oo anyway we assume from now on that the 
systems being considered are time-invariant and that the matrices R, Q are also 
independent of t. Thus, without loss of generality, we can restrict attention to 
the case of optimization on the interval [0, oo). The problem to be studied is, 
then: 

For the system 

x = Ax + Bu , 

and any given initial state x at time t = 0, find a control u> on [0, oo) 
such that the cost (possibly infinite) 

Joo(x,u;):= {w(t)'Ru(t) + Z(t)'QZ(t)}dt (8.53) 

Jo 

is minimized among all such controls. 

It is again assumed that R > and Q > 0. We define the Bellman function 
as in the finite interval case, and similarly the notions of optimal control and 
optimal trajectory. 

It will be shown that, under controllability assumptions guaranteeing that 
the cost can be made finite, a solution again exists and can be computed via 
an algebraic equation obtained as a limit of the Riccati differential equation. 
The assumption that R > insures that the problem is well-posed, in that no 
coordinate of the control is "free" of cost. Under the extra assumption that Q is 
positive definite rather than merely semidefinite, finiteness of the integral along 
an optimal trajectory will imply that £(£) must converge to zero, that is, the 
optimum system will be stabilized by the optimal control law. 

Consider the matrix Riccati equation (RDE) on the interval 

(-oo,0] 
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but with the final condition 

P(0) = 0. 

A solution P(-) exists on this interval. This is because a solution exists on each 
interval [a, 0] as follows from Theorem 37 (p. 364), and by the ODE Uniqueness 
Theorem these solutions must coincide on their common domain. 
Let II(t) := P(—t). Thus, H(t) satisfies for t > the equation 

tl = -UBR- 1 B'U + UA + A'U + Q, 11(0) = . (8.54) 

Equivalently, we may by time invariance consider the Riccati equation (RDE) 
on each interval [0, i], with final condition P(t) = 0; then H(t) is the same as 
the value P(0) for the corresponding solution. It follows that for each x € W l 
and each t > 0, 

x'TI{t)x = V t (0,x), 

where V T is the Bellman function for the optimization problem of minimizing 
(8.53) but only over the finite interval [0,r]. 

Take any pair fi > r > 0, and let v be optimal for the interval [0,/i] and 
initial state x. Then, 



x'IL(ij,)x = V„(0, x) = J(n, 0, x, v) 

= J(t,0,x,v\[Q,t)) + J Mt)'RHt) + m'Qm}dt 
> V t (0,x) = x'U(t)x. 

This holds for all x, so 

n(/x) > n(r) (8.55) 

in the sense of the partial ordering of symmetric matrices ( "P > Q" means that 
x'Px > x'Qx for all x). 

We prove next that, if the system is controllable, then there is a limit II for 
n(t) as t — » oo, and will use this fact in order to prove that there are optimal 
controls for the infinite time problem and that the optimal cost is given by the 
quadratic form x'Ux. 

Lemma 8.4.1 Assume that E is controllable and let R and Q be as above. 
Then the limit 

II := lim n(t) 

t — >oo 

exists. Moreover, II satisfies the algebraic Riccati equation 



nBR~ 1 B'n - IL4 - A'n - Q = (ARE) 



Proof. Fix any x € W 1 . We first remark that controllability implies that there 
is some u) such that 

Joo(x,uj) < oo . 
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Indeed, let ui\ be any control with 

0(l,O,a;,a;i)=O 

and consider the control u on [0, oo) that is equal to uj\ for t G [0, 1] and is 
identically zero for t > 1. Then, 

J 00 {x,w) = I {ui(t)'Rui(t) + Z(t)'Qi(t)}dt<oo. 
Jo 

It follows that, for each r > 0, 

s'n(T)x = y T (0,a;) <J"(t, 0,1, w| [0 ,r)) < Joo(x,w) . (8.56) 

Thus, 

{a;'n(r)a;} 

is not only nondecreasing in r (equation (8.55)) but it is also bounded above. 
Thus, 

lim x'H(t)x 

T — >00 

exists, for each fixed x. Let lij be the limit obtained when x = ei + ej, and U the 
limit for x = e%, where e\ is the ith canonical basis element in 1", i = 1, . . . , n. 
Then, for each pair i,j, using that H(t) is symmetric for each t: 

n(% = ejn(t) ei = i[(e, + e,)'n(t)(e l + ej) - e^n(i)e 4 - ejn(t) ej -] 

converges to 

n["ij ~ 'i ~ hi • 

So a limit II exists, as desired. Since II(£) satisfies (8.54), 

lim (d/dt)(x'U(t)x) 

t — ^oo 

also exists for each x, and hence this limit must be zero. It follows that 

x' [UBR^B'U - UA - A'U - Q] x = 

for all x. Symmetry of the matrix in brackets [. . .] implies that (ARE) must 
hold. I 

From equation (8.56), which holds for any control u>, we conclude that 

x'Wx < Joa{x, to) 

for all u>. When v is the closed-loop control on the interval [0, oo) corresponding 
to the feedback law 

k(t,x) := —Fx 
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with 

F := R- x B'\l 

and starting at £(0) = x, we have, substituting uj = v = — R~ 1 B'H^ into (8.53), 
that 

/>oo 

Joo(x,v)= / t(t)'(nBR- 1 B , iL + Q)i(t)dt. 

Jo 
On the other hand, 

(d/dt)(t(t)'iit(t)) = -^'(mjir^'n + Q)e(t) , 

because II satisfies (ARE). It follows that 

J x {x,v) = x'lix - lim £(i)'ll£(i) < x'lix < Joo(x,u) 

t — >oo 

for all uj (and, in particular, for v). Thus, 

• v is optimal; 

• x'Tix is the Bellman function for this problem; 

• J<x>{x-,v) < oo; and 

• limbec £(t)'II£(t) = 0. 

We now prove that v is the unique optimal control. Let 

a(t) := CW'nC(i) 

computed along any arbitrary trajectory (£, u>) on [0, oo) such that £(0) = x for 
which w/i/. Completing the square, one verifies just as in the finite-time case 
that the minimum of 

v' Rv + 2x'n[Ax + Bv] 

is achieved only at 

v= -R^B'lix. 

That is, 

uj'Ruj + a> (R^B'U^)' R (R^B'U^) + 2£'II (A - BR^B'U) £ 

for all t, and equality cannot hold almost everywhere since that would mean 
that uj = —R~ 1 B'H^, which we assumed was not the case. 

Using this as well as the fact that II satisfies (ARE), we conclude that 

a(t) > -u{t)' Ru{t) - £{t)'Q£{t) , 

with strict inequality on a set of nonzero measure. Integrating, there exists 
some t > such that 

x'\lx< J {w(t)'Roj(t) + t(t)'Q£(t)}dt + £(t)'II£(t) . (8.57) 

Jo 
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Consider z := £(r) and the control 

u)(i) :=u>(t + T),t>0. 

Since z'Wz is the value of the Bellman function at z, the last term in (8.57) is 
majorized by J oa (z,uj). By time invariance, 

J(t,0,x,u>) + Joo(z,u>) = Joo{x,u>) , 

which is therefore strictly larger than x'Wx. This shows that the control lu 
cannot be optimal, and the uniqueness statement follows. 

Theorem 41 Consider the infinite time problem defined by £,i?, Q, and as- 
sume that £ is controllable. Then there exists for each x £ R" a unique optimal 
control, given by the closed-loop control corresponding to the (constant) feedback 

F := -R- l B'U, 

where II is the solution of the ARE obtained by the above limiting process. Fur- 
ther, if in addition 

Q is positive definite, 

then 

II is also positive definite, 

and for any solution of the closed-loop system 

i=(A + BF)ti 

necessarily 

lim £(£) = . 

t — >oo 

Proof. We proved the first part above. Assume now that Q > 0. Once we 
prove that II > 0, as we proved already that 

for all trajectories, it will follow also that £(£) — * 0, as desired. 

Since II is the increasing limit of the matrices TI(i) , it will be sufficient to 
prove that H(t) is positive definite for all t > 0. Consider the solution II(-) of 
the differential equation (8.54). Fix any x ^ 0. Let f3(t) := x'TL(t)x. Then 

/3(0) = x'Qx > 

and (3(0) = 0, from which it follows that 

(3(t) > 

for all small t > 0. Since (3 is nondecreasing, it follows that x'H(t)x > for all 
t. I 
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Exercise 8.4.2 Show that Theorem 41 (p. 384) is valid if one replaces "E is 
controllable" by "E is asycontrollable." Point out exactly where in the proof 
the weaker hypothesis can be used. □ 

Given any controllable system E, we may always consider the optimal control 
problem with 

R=I : Q = I. 

Then Theorem 41 can be applied in this case, and we conclude in particular, 
using Proposition C.5.1: 

Corollary 8.4.3 If E is a controllable time-invariant linear continuous-time 
system over K = R, then there exists an m x n real matrix F such that A+ BF 
is a Hurwitz matrix. □ 

The above Corollary is, of course, also a consequence of the Pole-Shifting 
Theorem, which establishes a far stronger result. Next we consider a discrete- 
time analogue; its proof follows the same steps as in the continuous-time case. 

Lemma/Exercise 8.4.4 Consider the infinite time problem of minimizing 

DC 

J^x, w) := ^2{u(t)'Ru(t) + m'Qttt)} G K + |J{+oo} 

4=0 

for a discrete-time linear time-invariant system E. Assume that 

• E is controllable; 

• R is positive definite; 

• Q is positive semidefinite. 

Then there exists for each x G K n a unique optimal control, given by the closed- 
loop control corresponding to the (constant) feedback 

F := -{R + B'TIB)' 1 B'lLA , 

where II is the solution of the following discrete-time algebraic Riccati equation 
obtained as a limit of solutions of appropriate finite-time problems (cf. Exercise 

8.2.7): 



U = A'[U- UB(R + B'llB)- 1 B'Il}A + Q (DARE) 



Further, if in addition 

Q is positive definite, 

then 

II is also positive definite, 

and for any solution of the closed-loop system 

t = (A + BF)£ 
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necessarily 

lim £(£) = . 

t — ►OO 

holds. D 

Exercise 8.4.5 Prove, without using the Pole-Shifting Theorem: If E is a con- 
trollable time-invariant linear discrete-time system over IK = R, then there exists 
anmxn real matrix F such that A + BF is a convergent matrix. □ 

Numerical Computation of Solutions of ARE 

Solutions of the ARE can be obtained as limits of solutions of the differential 
equation or by other numerical techniques. One of the latter is based on the 
following considerations. 

As done earlier, we consider the Hamiltonian matrix: 

" =- (-« ~ B -a' B ') ■ < 8 ' 58 > 

The next problem shows that finding solutions of the Riccati equation (ARE) 
is equivalent to finding certain invariant subspaces of this matrix. In general, 
associated to any matrices A, N, L we may consider the following equation gen- 
eralizing (ARE): 

-ILLII - IL4 - ATI + N = (8.59) 

to be solved for an n x n matrix II. 
Exercise 8.4.6 Consider the map 

G n {R 2n ), 11^ col 



2 "l II — ,-,,1 f I 



where G n (R 2n ) is the set of all n-dimensional subspaces of K 2n (the "G" stands 
for Grassman manifold) and where "col" denotes column space. Prove the 
following facts: 

1. a is one-to-one. 

2. The image of a restricted to solutions of (8.59) is exactly the set of n- 
dimensional //-invariant subspaces of the form col I for some P. 

3. If II is a solution of Equation (8.59), then the restriction of H to a(II) has 
matrix A + LU in the basis given by the columns of _ 

4. The matrix II is symmetric iff a(II) is a Lagrangian subspace. (A La- 
grangian subspace V of R 2n is one for which x'Jy = for all x,y G V, 
where J is the matrix in Equation (8.33).) 
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Conclude that, if H is as in (8.58), the symmetric real solutions of the Riccati 
equation are in one-to-one correspondence with the Lagrangian i7-invariant sub- 
spaces of R 2n which have an onto projection into the first n coordinates. □ 

It follows from this exercise that in principle one can compute solutions of 
(ARE) by spectral analysis of H. In particular, if H has n eigenvectors V\ , . . . , v n 
with the property that, if we partition the 2n x n matrix (yi, . . . , v n ) as 

X 

Y 

with each of X and Y being an n x n matrix, then X is nonsingular. It follows 
that YX^ 1 is a (not necessarily symmetric) solution of the ARE. 

In general, there is more than one solution of the ARE, even for positive 
definite Q. However, as we shall prove below, there is only one positive definite 
solution II, which is then necessarily (for Q > 0) the one needed for the solution 
of the optimal control problem. 

We proved above that A + BF is Hurwitz when II is the solution of the ARE 
obtained as the limit of the II(t) and F = —R~ 1 B'T1. This property holds in 
fact for any positive definite solution of the ARE, as proved next. 

Lemma 8.4.7 Assume that II is a positive definite solution of the ARE and 
that Q is also positive definite. Then, defining F := —R~ 1 B'U, the closed-loop 
matrix A c \ = A + BF is Hurwitz. 

Proof. Manipulation of the ARE shows that 

A' cl U + UA ct = -Q - UBR^B'U . 

Thus, n is a positive definite solution of a Lyapunov matrix equation 

A^U + UA cl = Qx 

and Qi is negative definite, since it is of the form — Q — Q 2 with Q positive 
definite and Q2 semidefinite. We proved earlier that the existence of such a 
solution implies that A c i must indeed be Hurwitz. It is always instructive to 
provide alternative proofs, so we establish this directly as follows. 

Let A be any eigenvalue of A c i, and let v be any corresponding eigenvector. 
Then, 

A c iv = Xv , v* A' cl = Xv* . 

Multiplying the Lyapunov equation on the left by v* and on the right by v, 

KeA=I(A + A) = (I)^<0 
2 2 v U-V 

from which we conclude that A is Hurwitz. I 
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Exercise 8.4.8 (Infinite gain margin of LQ feedback.) As above, suppose that 
II is a positive definite solution of the ARE and that Q is also positive definite. 
Pick any p s [1/2, oo) and let F := — pR~ 1 B'H. Show that the closed-loop 
matrix A c i = A + BF is Hurwitz. □ 

The result in Lemma 5.7.18 is needed in the next proof. This states that the 
operator 

C : R" x " -> R nx ™ , £{X) := MX + XN . 

is invertible if both M and N are Hurwitz. 

Corollary 8.4.9 If £ is controllable and R, Q are both positive definite, then 
there is exactly one positive definite solution II of the ARE. 

Proof. Assume that two such solutions IIi,n2 exist. Equating the two corre- 
sponding equations, there results the equality MX + XN = 0, where 

M = (A-BR^B'^)' 
N = A- BR^B'IL-l 
X = Ili-IIa. 

Thus, X is in the kernel of the operator C introduced above. By Lemma 8.4.7, 
both M and TV are Hurwitz. It follows from Lemma 5.7.18 that C is one-to-one, 
so X = 0, as wanted. I 

Example 8.4.10 Take the system 

x = 3a; + 4m 
and the problem of minimizing 

/•oo 

/ x 2 + u 2 dt . 
Jo 

Then the (ARE) is simply 



167T 2 -67T- 1 = 0, 

and it has the two roots it = 1/2 and ir = —1/8. The unique positive one is the 
first, and this results then in the optimal feedback law 

u = —2x 

and the closed-loop system 

which clearly has a Hurwitz matrix. Alternatively, one may form the Hamilto- 
nian matrix 

' 3 -16' 
-1 -3 
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which has eigenvalues ±5 and corresponding eigenvectors 

I 8 ) and (l 

which, when taking the quotient of the second coordinate over the first, give 
again the two solutions 1/2,-1/8. □ 

Exercise 8.4.11 For the same system x = 3x + 4u consider instead the cost 



f 

Jo 



u 2 dt. 



Show that here there is also a unique positive definite solution of the Riccati 
equation, but this is not the one giving rise to the optimal feedback law. Explain 
why none of the results given earlier are contradicted by this example. □ 

The proof of Lemma 8.4.7 involved the fact that the Riccati equation can 
be rewritten as a Lyapunov equation for the closed-loop matrix. Another, basi- 
cally equivalent, relation between the two equations is that Lyapunov equations 
appear as partial differentials of the Riccati equation with respect to II. More 
precisely, if we think of the left-hand side of the (ARE) as defining a map 

p(A, B, Q, R, P) := PBR- l B'P -PA-A'P-Q 

whose domain is the set of all controllable pairs (A, B) and positive definite 
symmetric Q, R, P (as an open subset of a suitable Euclidean space), then, for 
any IIo for which 

p(IIo) = 

it follows that 

p(U + P) = - [PA cl + A' cl P] + o(\\P\\) (8.60) 

for all P. By Corollary 8.4.9 there is, for each (A,B,Q,R) as above, a unique 
solution of (ARE), that is, there is a function a so that 

p(A,B,Q,R,a(A,B,Q,R)) =0 

identically. 

Exercise 8.4.12 (Delchamp's Lemma) Prove that a is differentiable (in fact, 
real-analytic) by showing that equation (8.60) holds and applying the Implicit 
Function Theorem. Conclude that there exists a C 1 (and even analytic) function 

: S^ m -» R" xm 

that assigns to each controllable system (A, B) a feedback matrix F = f3(A, B) 
such that A + BF is Hurwitz. □ 
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Finally, for the filtering problem too, when the system is time- in variant and 
R, Q are also independent of t, it is useful to consider the limit of the solution II 
of (FRDE) as t — > oo. The gain matrix L = —HC'Q is often called the steady- 
state Kalman gain and the filter equation (8.52) with this gain (now itself a 
time-invariant observer) the steady-state Kalman filter. 

Exercise 8.4.13 Show that, if (A, C) is observable, then II := lim^oo II(i) 
exists and that it satisfies the dual algebraic Riccati equation 

au + ua 1 - nc'Qcn + brt^b' = o . 

{Hint: Use duality.) □ 

8.5 Nonlinear Stabilizing Optimal Controls 

Theorem 41 shows that the solution of the steady-state linear-quadratic optimal 
control problem can be expressed in the feedback form u = —RB'Tlx, where 
II solves the Algebraic Riccati Equation, and, furthermore, this feedback law 
stabilizes the system. We discuss briefly a partial generalization, to nonlinear 
continuous-time time-invariant systems, of these facts. Just as the material in 
Section 8.4 represents an infinite-time version of the finite-time optimal control 
problem in Section 8.2, the results discussed here are closely related to those 
in Section 8.1. However, we derive the results in a direct and self-contained 
manner. 

In this section, we work with a continuous-time time-invariant system 

x = f(x,u) 

(over K = K). We assume that 0e I. For each state x G X, we let £l x denote 
the class of all "stabilizing" controls, meaning by definition the set of locally 
essentially bounded 

lu : [0, oo) — » U 

with the property that the solution £(t) = (f>(t, x, lu) is defined for all i > (that 
is, to is admissible for x), and 

lim £(£) = . 

t — ►OO 

The objective is to minimize, over all lo G f^, the cost 

f-OC 

Joo(x,uj) := / q{C{s) 7 uj{s))ds, 
Jo 

where (!;,lu) is the trajectory with £(0) = x. The function 

q : X x IX -> M.+ 
is assumed to be continuous. 
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In all the results to be described, we assume given a continuously differen- 
tiable function 

V : X^R, V(0) = 0. 

This function will play the role of a value or Bellman function, as well as a 
Lyapunov function for a closed-loop system. Given such a function V, for each 
(x, u) G X x 11 we denote 

V(x,u) := VV(x)-f{x,u). 

Proposition 8.5.1 Suppose that 

V(x, u) + q(x, u) > for all (x,u) G X x IX (8.61) 

and that a path (£,,oj), with £(0) = x° and u G fi x o, is so that 

V(Z(t),w(t)) + q(€{t),w{t)) = for almost all t > . (8.62) 

Then, V(x°) is the optimal cost from x , and uj is an optimal control, i.e.: 

V{x°) = Jooix ,^) = min Jooix ,^). (8.63) 

Proof. Consider any v G fJ x o, and let £(£) := (j)(t,x ,v). Since V {(,(€)) is 
absolutely continuous, we can write, using (8.61): 

t d 



V(((t))-V(x u ) = J -V(((s))ds = J V(C(s),u(s))ds>-l q(((s),v(s))ds 



for all t > 0, from which we conclude that 
„o 



V(x°) < V(((t)) + / q(((s),v(s))ds. 
Jo 

Taking limits as t — > oo, and using £(£) — » and continuity of V at zero, we 
obtain 

V(X°) < / «(C(«), *(*))<*« = Joo(x°,v). 



The same arguments, for v = lo and using (8.62), give V(x°) = Joo{x° ,u>). I 

From this, we conclude immediately that the (infinite-time) Hamilton- Jacobi 
equation is sufficient for optimality: 

Corollary 8.5.2 Suppose that k : X — » IX is so that 

\/x,u V(x,k(x)) + q(x,k(x)) = min < V(x, u) + q(x, u) > = (HJB) 

and that a path (£,w), with £(0) = x° and w G £l x o, is so that w(£) = £;(£(£)) for 
all £. Then, V(x°) is the optimal cost, and u) is an optimal control, from x°. D 
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Theorem 42 Let k : X — » II be locally Lipschitz and so that (HJB) holds. 
Assume that X = W 1 , V is proper and positive definite, and q(x, u) > whenever 
x ¥" (f or a M u )- Then, for each state x°, the solution of £ = /(£,&(£)); 
£(0) = x° exists for all t > 0, the control co(t) = k(^(t)) is optimal, and V(x ) 
is the optimal value from x° . Moreover, V is a global Lyapunov function for the 
closed-loop system. 

Proof. Solutions exist locally, since k is locally Lipschitz. Since V(x, k(x)) = 
—q(x,k(x)) < whenever x ^ 0, V is a Lyapunov function for the closed-loop 
system. In particular, a solution exists as stated, and its limit is zero as t —* oo, 
so w G Cl x o. The claim now follows from Corollary 8.5.2. I 

We now specialize to C 1 control-affine systems 

in 

x = f{x) + ^2u i g i (x) = f(x) + G{x)u 

as in Equation (2.34) (but writing here / instead of go)) with X = R n , U = 
R m , and /(0) = 0. We suppose from now on that V is twice continuously 
differentiable, in addition to being positive definite and proper, and that 

q(x,u) = u'R(x)u+Q(x) 

where Q is a continuous function, positive for all x ^ 0, and R is a symmetric 
n x n matrix of C 1 functions of x, positive definite for each x. We denote 

L G V(x) := VV(x)G(x) = (L gi V(x), . . . ,L 9m V(x)) 

and introduce the following algebraic equation: 

Vi Q{x) + L f V{x) - l -L G V{x)R{x)- 1 {L G V{x))' = 0. (8.64) 

Corollary 8.5.3 Assume that (8.64) holds, and define 

k(x) := -^Rixy^LcVix))' . (8.65) 

Then, the conclusions of Theorem 42 hold. 

Proof. Just as in Remark 8.1.10, we may complete squares to show that (8.65) 
minimizes the Hamilton- Jacobi expression, i.e., in this special case: 

k(x) = argmin {LrV(x) + LqV(x)u + u' R(x)u + Q(x)} . 

u 

Note that k is locally Lipschitz, in fact C , because V was assumed to be C 2 . 
We are only left to verify the "= 0" part of (HJB), i.e. 

k{x)' R{x)k(x) + Q{x) + L f V{x) + L G V{x)k(x) = 0. 
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But this is simply (8.64). I 

Note that an even more special case is that when R(x) = I for all x. Then 

k(x) := - l -(L G V{x))' , 

which happens to be (up to a factor 1/2) the stabilizing feedback used in damp- 
ing control (cf. Lemma 5.9.1). Equation (8.64) reduces, in this case, to: 

1 m 

\/x Q(x) + L f V(x) - -J2lL gi V(x)} 2 = 0. 

Exercise 8.5.4 Show that for linear systems and quadratic V, Equation (8.64) 
reduces to the Algebraic Riccati Equation (ARE) given in Lemma 8.4.1. □ 

Exercise 8.5.5 Suppose that we drop the requirement that Q be positive def- 
inite, asking merely semidefiniteness, but we add, instead, the assumption that 
k, defined by formula (8.65), globally stabilizes the system. Show that we can 
still conclude that V is the value function and that the feedback u = k(x) gives 
the optimal controls. □ 

Exercise 8.5.6 Consider the following system (with n = m = 1): x = xu, and 
take the problem of minimizing 



4 + W dt 



among all controls making x(t) — > as t — > oo. Show that there is a solution, 
given in feedback form by a quadratic feedback k(x) = —ex 2 , for some c > 0. 
(Hint: Use as a guess a quadratic V(x) = px 2 . Find the appropriate p by 
solving the HJB equation.) □ 

Exercise 8.5.7 Generalize Exercise (8.5.6) to x = uAx, a system of dimension 
n and scalar inputs, with A having all eigenvalues with positive real part, and the 
cost q(x,u) = ^(x'Qx) 2 + \u 2 , where Q is a positive definite constant symmetric 
matrix. Show that also here one obtains a quadratic feedback solution. □ 

Exercise 8.5.8 (Infinite gain margin of LQ feedback; see also Exercise 8.4.8.) 
Suppose that V is a C 2 function which satisfies (8.64). Pick any p € [1/2, oo) and 
let k(x) := -pRix)" 1 (L G V{x))' . Show that x = f{x) + G{x)k{x) is globally 
asymptotically stable with respect to the origin. □ 

Exercise 8.5.9 Provide an example of a smooth system x = f(x, u) and a 
smooth cost q(x,u) so that (1) for each x° there is some uj G O x o so that 
Joo{x°,uj) < oo, but (2) the conclusions of Theorem 42 are false. In fact, show 
that any no-drift system x = G(x)u with rank G(0) = m < n which satisfies the 
Lie Algebra Accessibility Rank Condition, together with any g, gives rise to such 
a counterexample. (Hint: Use Exercise 5.9.20, noting that k given by (8.65) 
must be smooth.) □ 
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Exercise 8.5.10 Find the value function V and the optimal feedback solution 
for the problem of minimizing L x 2 + u 2 dt for the scalar system x = x 2 + u. 
{Hint: The HJB equation is now an ordinary differential equation, in fact, a 
quadratic equation on the derivative of V .) □ 

Exercise 8.5.11 Suppose that LfV(x) < for all x and that x = f(x) + G(x)u 
is globally stabilized by u = — (\7V(x) ■ G{x)) , as in Proposition 5.9.1. Show 
that u = k{x) is an optimal feedback, and V is the value function, for some 
suitably chosen cost. {Hint: Let Q(x) := — LfV(x) + \ LqV(x) (LqV(x)) , 
which gives (8.64) for which Rl Use Exercise 8.5.5.) □ 

8.6 Notes and Comments 

Dynamic Programming 

The dynamic programming approach gained wide popularity after the work of 
Bellman; see, for instance, [41]. Almost every optimization or optimal control 
textbook contains numerous references to papers on the topic. 

The Continuous-Time Case 

A central problem in optimal control is the study of existence and smoothness 
properties. The question of existence often can be settled easily on the basis of 
general theorems; what is needed is the continuous dependence of J on ui, with 
respect to a topology on the space of controls which insures compactness. For 
bounded inputs and systems linear on controls, Theorem 1 (p. 57), part 2 (ii) , 
together with compactness in the weak topology, achieves this purpose. For 
more general systems, analogous results hold provided that one generalizes the 
notion of control to include "relaxed controls." See, for instance, [134], [188], 
[266], or any other optimal control textbook; here we only proved results for a 
very special class of systems and cost functions, and for these existence can be 
established in an ad hoc manner. 

Continuity of controls, and smoothness of Bellman functions, are a much 
more delicate matter. It turns out that many problems of interest result in a 
nonsmooth V. The regular synthesis problem in optimal control theory studies 
generalizations of the results that assumed smoothness to cases where V is 
piecewise smooth in appropriate manners. See, for instance, [134] for more on 
this topic, as well as [50] and [386]. An alternative is to generalize the concept 
of solution of a partial differential equation: The notion of viscosity solution 
provides such a possibility, cf. [102]; or the related approach based on proximal 
subgradients can be used, cf. [93]. 

(Deterministic) Kalman Filtering 

A purely deterministic treatment was pursued here so as not to require a large 
amount of preliminary material on stochastic processes. On the other hand, 
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a probabilistic study is more satisfactory, as the cost matrices have a natural 
interpretation and formulas can be obtained that quantify the accuracy of the 
estimator (its covariance is given in fact by the solution to the Riccati equation) . 

There are excellent textbooks covering Kalman filtering, such as [108] or, 
for more of an engineering flavor, [16] and [263]. A good reference for practical 
implementation issues, as well as a detailed derivation in the discrete-time case, 
is provided by [153]. A very elegant discussion of the discrete-time problem is 
given by [86], from the point of view of recursive least-squares. 

Much has been done regarding nonlinear filtering problems; for some of the 
theoretical issues involved see, for instance, [209], [293], [312], and the references 
therein. 

Historically, the Kalman filter, introduced for discrete-time in [214] and for 
continuous-time in [229], appeared in the context of the general problem of 
"filtering" a signal corrupted by noise. As compared to older Wiener filtering 
ideas, the Kalman filter modeled the signal of interest as the state trajectory of 
a linear system driven by white noise. This allowed a highly computable method 
of solution as well as extensions to nonstationary processes (time- varying state 
models) and nonlinear systems. 

The optimal estimation and optimal control problems can be combined into 
the stochastic optimal control problem of minimizing a cost criterion for a system 
such as the one in Exercise 8.3.9, by suitable choice of controls u, on the basis 
of noisy observations; the solution to this "LQG" problem — "linear quadratic 
Gaussian problem," when all noises are assumed to be Gaussian processes — can 
be obtained from the solutions of the two separate deterministic linear quadratic 
and linear filtering problems, in much the same fashion as output stabilizers were 
obtained by combining observers and optimal state feedback laws; see [16] and 
[263], for instance. 

Linear Systems with Quadratic Cost 

The treatment of the Linear Quadratic Problem started with Kalman's seminal 
paper [215]. The literature generated by this problem is immense, as many vari- 
ants and numerical approaches have been tried. Two excellent texts that deal 
with such issues are [16] and [263]. The paper [423] discusses many important 
issues about Riccati equations, including the indefinite case, of interest in game 
theory as well as in modern H^ optimization. 

The infinite-dimensional linear case also has been studied in detail; see, for 
instance, [113] and [133]. 

Infinite-Time Problems 

The computation of solutions to the ARE via Hamiltonian matrices was first 
suggested by [288] and [322] for the case of distinct eigenvalues; see [212], Section 
3.4, for a discussion and many references to numerical techniques associated to 
this approach. 
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The result in Exercise 8.4.12 is from [112] (see also [64]). 

It is interesting but not very surprising that quadratic problems for linear 
time- invariant systems give rise to linear solutions (a linear feedback law). When 
other criteria are used, or when "robust" design is desired, nonlinear or time- 
varying controllers may be superior, even for linear time-invariant systems. This 
type of question is explored in [244], [245], and related papers. 

Substantial activity has taken place recently on the topic of H^ optimization, 
dealing with a different optimization criterion than the linear-quadratic problem. 
This criterion, which corresponds to the minimization of an operator norm from 
external "disturbances" to outputs, is of great engineering interest. A book 
reference is [145], and recent research, resulting in a reduction to a Riccati 
equation problem, is surveyed in [124]. That reference also shows how the linear- 
quadratic problem can be viewed as a minimization problem in an input/output 
context. The norm itself can be computed numerically; see, for instance, [59]. 

Nonlinear Stabilizing Optimal Controls 

The main sense in which the nonlinear results given in Section 8.5 differ from 
the linear case is in the need to assume that V exists. For linear systems, the 
ARE always has a solution (under the appropriate controllability conditions) . In 
order to obtain necessary and sufficient results, one must introduce generalized 
solutions of various types (viscosity, proximal); see for instance [93] and the 
many references provided therein. 



Chapter 9 

Optimality: Multipliers 



As described in the introduction to Chapter 8, an alternative approach to opti- 
mal control relies upon Lagrange multipliers in order to link static optimization 
problems. In this chapter, we provide a brief introduction to several selected top- 
ics in variational, or multiplier-based, optimal control, namely: minimization of 
Lagrangians (and the associated Hamiltonian formalism) for open input-value 
sets, the basic result in the classical Calculus of Variations seen as a special 
case, some remarks on numerical techniques, and the Pontryagin Minimum (or 
Maximum, depending on conventions) Principle for arbitrary control- value sets 
but free final state. The area of nonlinear optimal control is very broad, and 
technically subtle, and, for a more in-depth study, the reader should consult the 
extensive literature that exists on the subject. 

9.1 Review of Smooth Dependence 

It is worthwhile at this point to review some of the main conclusions of Theo- 
rem 1 (p. 57). Consider the system 

x = f(x,u), 

under the standard assumption that / : X x IX — > X is C 1 in x, and that /, 
as well as its Jacobian matrix of partial derivatives f x with respect to x, are 
continuous on x,u (cf. Equations (2.22 and (2.23)). The state-space is an open 
subset X of R™, and II is a metric space. These conclusions characterize the 
partial derivatives of the final value of the solution, that is x(t), with respect to 
the initial state x(a) and the input u(-) applied during an interval [er, t\. Here 
we restate the relevant facts in terms of Jacobians with respect to x(a) and 
directional derivatives with respect to u(-); Theorem 1 (p. 57) provides a more 
general statement concerning joint continuous differentiability. 

Recall that C\l(ct,t) (or £^(<t,t), in the special case U = M. m ) denotes the 
set of measurable essentially bounded maps w : [a, r] — » It, where "essentially 

397 
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bounded" means that there is some compact subset C = C u C IX such that 
Lo{t) G C for almost all t G [a, r], and we say that an input lu G C\i{u,t) is 
admissible for the state x° if the solution of the initial value problem 

i(t) = fM)Mt)), e(^) = ^° 

is well-defined on the entire interval [a, r], and we denote, in that case, 

</>(-, a,xV) = £(•)• 

For future reference, we now summarize the main needed facts. 

Corollary 9.1.1 (of Theorem 1 (p. 57)). Let lu G Cu{c,t) be admissible for 
the state a: , and write x(t) = <p(t,a, x°,u>). Denote 

A(t) = f x (x(t),co(t)) (9.1) 

for each t G [a, r], and let $(•, •) be the fundamental solution matrix associated 
to A{-): 

^fe) = A(t)$(t,r), *(r,r)=J (9.2) 

for each t,r G [a, r]. 
Consider the map 

8 : i h <p(r, a, x, lu) . (9-3) 

Then, is well-defined and continuously differentiable in a neighborhood of x° , 
and its Jacobian evaluated at x° is 

9, (a; ) = $(t,ct). (9.4) 

Suppose that, in addition, the control- value set U is an open subset of K m 
and that / : X X "U — > 1" is continuously differentiable, and let 

B(t) = f u (x(t)Mt)) ■ (9-5) 

Take any fixed input n G C\i{a, r), and consider the map: 

/3 : (-/i ,/i )^K n : /i i-» 0(r, ct, a; ,S + /i/x) . (9.6) 

This map is well-defined for some ho > 0, is continuously differentiable, and 
dp 



dh (0) = J *(T,8)B(a)n(8)d8. (9.7) 

More generally, if /ii, . . . , \x r G Cu{c, t), the r-parameter map 

P(h\, . . . , h r ) := <f>(r, a, x Q , lu + h\(ii + . . . + h r [i r ) . 
has Jacobian 



&(0) = / $(T, S )B( S ) W ( S )d S ,.., / *(T,*)B(*K(*)d« . □ 



9.2. Unconstrained Controls 399 

9.2 Unconstrained Controls 

When control-value sets are open, various facts about optimal control can be 
easily derived by finding critical points of appropriate functions, with the use, 
if necessary, of Lagrange multipliers. In this section, we pursue this line of 
thought. So we take here IX C W 71 open and f : X x It — > M™ continuously 
differentiable. 

Assume given three C 1 functions 

q:XxU^R, p : X ^ K , K : X -» W , 

to be used, respectively, to denote the instantaneous cost along trajectories, a 
cost on final states, and a set of r constraints on final states. Also fix an initial 
state a; G X and a time interval [a, r] on which the optimization is to be carried 
out. The cost function associated to this problem is as follows: 



J(u) := / q(x u (s),u>(s))ds + p(x u (t)) 



where x UJ (-) := </>(•, a, x° , io) and we define J{uj) := +oo if lu is not admissible 
(for the initial state x ). The optimization problem of that of finding 



min{J r (cj) | uj g Lu{o~,t) admissible, K(x oj (t)) = 0} 



For example, a quadratic cost along trajectories, as in the classical linear- 
quadratic problem, would be one of the form q(x,u) = x'Qx + u'Ru. A problem 
in which we want to impose that the final state be a desired target x l is included 
by letting K(x) = x — x l ; on the other hand, K = covers the problem of 
minimizing with no final state constraints. Other special cases of interest, to 
be discussed separately, are those in which there is no final cost (i.e, p = 0) or 
there is no "Lagrangian" (q = 0). 

Definition 9.2.1 The control lj is optimal for x° if it is admissible for x° 
and has the property that J{u>) < Jiy) for all those admissible u> for which 
K(x u (t)) = 0. □ 

Since the domain of J is open, when K = first-order necessary conditions 
for optimality are obtained by asking that any u> minimizing J must be a 
critical point of J , i.e. that we have a zero differential at that input: J* [to] = 0. 
In the case in which constraints are present, the Lagrange multiplier rule can 
be employed. This program can be carried out in elementary terms, with no 
recourse to the Lagrange multiplier rule in infinite-dimensional spaces; we do so 
next. 

Recall the classical Lagrange multiplier rule, applied to a differentiable func- 
tion / : O — > R, where O is an an open subset of Euclidean space: if / 
achieves a local minimum (or maximum) on O subject to the differentiable 
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constraints g\{y) = ... = g r (y) = at the point y, and if the Jacobian G*(y) of 
G = (<7i, . . . , g r )' at y has rank r, then there exists some r-vector v so that the 
function / + v'G has a critical point at y. If, instead, the Jacobian has rank 
less than r, then the rows of G*(y) are linearly dependent, and therefore there 
is some nonzero r-vector v so that v'G*(y) = 0, which means that O.f + v'G has 
a critical point at y. One can summarize both cases in one statement, namely, 
that there are a scalar vo G {0, 1} and a vector v G R™, not both zero, such that 
the function v^j + v'G has a critical point at y. (Example: The minimum at 
y = of f(y) = y under the constraint y 2 = 0; here one must take vq = 0.) 
We next introduce, for our control problem, a condition corresponding to the 
Jacobian G*(y) having rank r. 

Let ui € ££?(<7, t) be admissible for x°, write x{t) = x%(t) = <fi(t, a, x°,u>), 
let A(-) and B(-) be as in Equations (9.1) and (9.5), and let $ be as in Equa- 
tion (9.2). 

With respect to the given function K : X — > W, we will say that the lin- 
earization along u is output controllable if 

K.(x(t)) J $(r,s)B( S ) M (s)ds M G £~(<r, r)| = M r . 

Equivalently, there must exist r inputs Hi, . . . , /j, r so that the r vectors 

#„(£(t)) I ^{T, S )B{ S ) f l 1 { S )ds, ... , /C.(x(t)) /" *(T,8)B( 8 )lir(8)ds 

(9-8) 
form a basis in M r . Observe that, in particular, K*(x(t)) G K rx ™ must have 
rank r, and the vectors in R™ 

®(T,s)B(s)m(s)ds, ... , / ®(t, s) B(s) Hr(s) ds\ (9.9) 

must be linearly independent. If there are no final-state constraints (r = 0, 
K = Q), linearized output controllability holds by definition. (The condition will 
only play a role when using Lagrange multipliers.) 

Remark 9.2.2 One may interpret linearized output controllability in terms of 
the time-varying linear system that has the above A(-) and B(-) in its state 
equations, as well as an output function given by the evaluation of the Jacobian 
of K along the given trajectory: C{t) := K*(x(t)). The condition says that 
every possible output value is achievable at time r. □ 

We now derive a necessary condition for an extremum. To make the proof 
more transparent, it is convenient to take first the special case when q = 0, and 
later to extend the result to the general case. 

Lemma 9.2.3 Suppose q = 0. Assume that u> € £u{ <J , T ) is optimal for x°. 
Then, there exist a scalar vq € {0,1} and a vector v G K r , not both zero, so 
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that the solution A : [a, r] — » R" of the adjoint equation A(i) = — ^4(£)'A(£) with 
final value 

A(r) = (v p4x(t)) + v'K4x(t)))' 

satisfies 

X(t)'B(t) = 

for almost all £ € [it, t] . If, in addition, the linearization along uj is output 
controllable, then one may take Vq = 1. 

Proof. We first treat the case of no final-state constraints (r = 0, no "if", and 
output controllability is by definition satisfied). Take any fixed fi G Cu(&, t), 
and consider the map /3(/i) := 0(t, ct, a; , 5 + /i/i), as in Equation (9.6). As 

p(/J(0)) = p(s(r)) = J(w) < J(w + hn) = p(0(h)) 
for all ft near zero, and p o /3 is differentiable, it follows that 

= ^T^ (0) = p * (i(r)) ^ (0) = ^P*(x(r))^(T,s)B( s ) fi (s)ds. 

Since /x was arbitrary, this means that 

P*(2?(t))$(t, *)£(*) = 

for almost all t. Since 

^?M = -$(r,t)^), (9.10) 

A(t) = (p,(x(r))$(r,t))' solves A(t) = -A(i)'A(t) and has A(r) = p„(z(r))'. 

Now we treat the case when there are constraints (r > 0). There is some 
integer 1 < £ < n and inputs (X\, . . . , \xn so that the following property holds: 
for every jj, G £ix(cr, r), the vector J $(t, s)B(s)li(s) ds is a linear combination 

of 



*(T,8)B(8)fil(8)d8, ..., $(T,s)B(s) l jLi(s)ds\ . (9.11) 

To see this, we consider the subspace 

ix e C u {<?,t) \ c 



$(t,s)B(s)/x(s) ds 

(which is the reachable set from the origin for the linearized system along (x, ui)): 
if this space is {0}, we pick £ := 1 and take any input as (Xi; otherwise, we let £ 
be the dimension of the space and pick a set of inputs so that the states in (9.11) 
form a basis. 

Consider now the map 

(3(hi, . . . , h e ) := </>(t, a, x" , lo + hi/xi + . . . + htui) , 
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which is defined on some open neighborhood O of in Mr. We consider the 
compositions p o j3 and K o f3 as maps O —* M. Using h to denote vectors 
(hi, . . . , hi)' , we have that 

(po/3)(0) = p(x(r)) = J{w) < J(u + hifii + . . . + hint) = (p°P)(h) 

for all those h G O for which (K o f3)(h) = 0. Furthermore, the Jacobian of 
K o j3 at /i = is iT*(a;(r))/3*(0), and a formula for the Jacobian /3*(0) is given, 
in turn, in the last part of the statement of Corollary 9.1.1. It follows that the 
columns of the r x £ Jacobian matrix (K o /3)*(0) are given by the vectors 



K*(x(t))I $(T,s)B(s)n 1 (s)ds, ... , K,(x(t)) I $(T,s)B(s)i M (s)ds 

(9-12) 
The Lagrange multiplier rule (in the form discussed earlier) provides a nec- 
essary condition for a local extremum of po (3 at h = subject to the constraint 
(K o P)(h) = 0: There exist a scalar vq € {0,1} and a vector v G R r , not 
simultaneously zero, such that 

d(v poP + v'KoP) ^ 

— (0) = , « = !,.. .,£. 

dhi 

That is, 

= {^oP+^K)4x{t)) f 3>{T,s)B{s)m{s)ds, i = l,...,£. 



Pick any /i 6 £ii(er, t). Since, for each /!, J < 1 ) (t, s)B(s)/j,(s) ds can be expressed 
as a linear combination of the vectors J &(t, s)B(s)ni(s) ds, we conclude that 
$1 {"OP + v' K)*(x(t))$(t, s)B(s)n(s) ds = for all u e £ u (cr, r) . So 

(i/ p+i//ir).(*W)*(r,t)B(t) = 

for almost all t. As with the case r = 0, Equation (9.10) says that X(t) = 
{{vqP+ v' K)*{x{t))<&{t, t))' solves \(t) = -A{t)'\{t) and has the desired final 
value. 

Finally, assume that r > and that the output controllability condition 
holds. Then, there exist inputs /j,i, . . . ,\x r so that the vectors in Equation (9.8) 
span W ' . Thus, also the vectors in Equation (9.9) are linearly independent, so 
we can add if necessary inputs /i r +i, • • • , A*£ (for some integer r < £ < n) so 
that Hi, . . . , fie form a generating set as above, and we use these inputs when 
defining (3. Since the first r columns of the r x £ matrix (K o /3)» (0) are given by 
the vectors in Equation (9.8), (K o /3)*(0) has full rank r. Then, the Lagrange 
rule applies in the form usually stated, for nonsingular constraints, and one may 
indeed pick Uq = 1. I 
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Remark 9.2.4 One interpretation of the Lemma is as follows. Let 1Z de- 
note the set of states reachable from the origin in the interval [cr, r] , for the 
linear system defined by A(t),B(t) (the linearization along (u3,af)); that is, 
K = {/J $( T , s)B(s)/j,(s) ds, n e £^(cr, r)}. Then, the vector (v p+v>'K)*(x(t)) 
must be orthogonal to the subspace TZ C R™. D 

Now we treat the general case, when there is a nonzero Lagrangian term q 
in the cost function. At this point, it is useful to introduce the Hamiltonian 
function associated to the optimal control problem, defined by 



H{x,u,7] ,r]) := rj q(x,u) + rj'f(x,u) 



and seen as a function of X x It x R x R n — > R. 

Theorem 43 Assume that uj € Cu(ff, t) is optimal for x° . Then, there exist a 
scalar v {) g {0, 1} and a vector v G W , 

{y Q ,u) ± (0,0), 

so that the solution A : [a, r] — » R" of the final-value problem 

A(i) = -m«x(x(t).w(*))' " ^W'A(i), A(r) = ( !/0 ^(i(T)) + .X(i(T))) , 

(9.13) 
is .so that 

™(x(t)Mt),vo,\(t)) = (9-14) 

for almost all t € [a, r] . //, m addition, the linearization along ui is output 
controllable, then the same conclusion holds with vq = 1. 

Proof. We reduce this problem to the one treated in Lemma 9.2.3. For that 
purpose, we introduce the system &fr = f#(x#,u) with state-space R x X C 

R™ +1 which, when writing x# = I , has equations 

x = q{x,u) 
x = f(x,u) 

(with same input value space U). We also think of K as a function on R x X 
that depends only on the last n coordinates, that is, we let 

K*{x*) := K{x). 

It is clear that the output controllability condition holds for the extended system 
if it holds for the original one. 

To define a minimization problem for the extended system, we let 

p* ) := xq + p{x) and q^ = , 
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and consider the initial state 

x°* - ( ° 

X - [x° 

We write J# for the associated cost function. Observe that an input uj is 
admissible for x° if and only if it is admissible for a; because, once that the 
equation for x has been solved, Xq{t) can be obtained by integration: xq(t) = 
J q(x(s),u)(s)) ds. Moreover, the definitions imply that 

J*(uj) = J{uj) 

for all uj. Therefore, 2 is a minimizing input for Jj&, and we can thus apply 
Lemma 9.2.3 to it. 

Let vq £ K, v G W , and A* : [it, t] — > R" +1 be as in the conclusions of the 
Lemma, for the extended system. Observe that 

W \q x {x(t),uj(t)Y A{t)' J W V B(t) 

As pf = (1 , p*) and K* = (0 , if*), the final condition on A* is: 

A#(r) = (^opf (x(r)) + v'K* (x(t)))' = ( , ,_, *° ljf ,_, ^/ 

We partition 

^ = {%) ■ 

Comparing the last n coordinates provides the final value A(r) shown in (9.13). 
Moreover, the differential equation A # (£) = —A#(t)\#(t) gives that Ao = vq 
and that A satisfies the differential equation in (9.13). Finally, X&(t)' B#(t) = 
implies that 

= \*(t)'B*(t) = Va q u (x{t),dj{t)) + \(t)'B(t) = H u (x(t),Z(t),v ,\(t)) 

for almost all i. I 

Theorem 43 provides a necessary condition for arbitrary local extrema of 
J . As in elementary calculus, it cannot distinguish between (local) minima and 
maxima of J '; higher-order derivatives may be used for that purpose (see also 
the maximum principle, below). 

Exercise 9.2.5 Show that, along any optimal trajectory for which uj is differen- 
tiable, the function t t— > H(x(t), u>(t), ^o, A(i)) must be constant. (The constancy 
of 7i(x(t), uj(t), vq, A(t)) holds, in fact, even if uj is not necessarily differentiable, 
but the proof is a bit harder.) □ 
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Exercise 9.2.6 Provide a version of the Theorem that applies to time-varying 
systems (and time-dependent costs), that is, x = f(t,x,u), and q = q(t,x,u). 
You may assume that / and q are continuously differentiable. (Hint: Easy from 
the Theorem; you only need to extend the system in a certain way, and then 
"unpack" the resulting equations.) □ 

Exercise 9.2.7 A problem for which q = is often called a Meyer problem of 
optimal control (this follows similar terminology used in the calculus of varia- 
tions). A Lagrange problem, in contrast, is one for which p = 0. (The general 
problem which we considered, incidentally, is sometimes called a "Bolza" prob- 
lem.) The main idea which was used in deriving Theorem 43 from Lemma 9.2.3 
was the observation that every Lagrange problem can be recast as a Meyer 
problem. Show that, conversely, every Meyer problem can be reformulated as 
a Lagrange problem, in such a manner that optimal controls are the same, for 
every possible initial state. (You may assume that p € C 2 .) □ 

Exercise 9.2.8 An optimal control ui is abnormal if it is impossible to obtain 
the conclusions of Theorem 43 with v§ = 1. That is, for each A(-) satisfy- 
ing (9.13) and (9.14), necessarily v§ = 0. Show that x = u 2 , x(0) = x(l) = 0, 
IX = [—1, 1], q(x,u) = u, p = 0, leads to an abnormal optimal control. □ 

One particular case of interest for the Theorem is as follows. The notations 
are as earlier. 

Corollary 9.2.9 Assume that ui is optimal for x Q , that the linearization along 
lu is controllable, and that r = n and the map K : X — ► W 1 has a nonsingular 
Jacobian K*(x(t)) at the final state. Then, there exists a solution A : [o~,t] — > 
1™ of the adjoint equation 

X(t) = -q x (x(t),u(t))' - A(t)'\(t) (9.15) 

so that Ti u (x(t),u)(t), 1, A(t)) = for almost all t € [a, r]. 

Proof. We need only to check the output controllability condition. But 

K.{x{t)) |y ^{T,s)B{s) l x{s)ds h€£%(<t,t) 

holds because the first matrix is nonsingular and the space {. . .} is, by assump- 
tion, equal to M™. I 

This corollary applies, in particular, when the final state is specified to be a 
given x = x i . We let in that case K(x) = x — x ! , so K has full rank at every state 
(Jacobian is the identity). Observe that, in contrast with Theorem 43, we did 
not specify a condition on A(r). There is no information to be gained by asking 
that A(t) = (p*(x(r)) + v 1 K*(x(t)))' for some v, because of nonsingularity of 
K,{x). 
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Writing as Hamiltonian Equations 

An elegant way of packaging the conclusions of Theorem 43 is by means of 
Hamiltonian equations, as follows: if lo is optimal for a; , then there is a solution 
{x{), A(-)) of the following two-point boundary- value problem: 

x = 7ix(x,cj, vo, A)' , x(a) = x° , K(x(t)) = 

A = -n x (x,u,uo,\)' , A(r) = (vop*(x(t)) + v 1 X(x(r)))' 

and, along this solution, it must hold that Ti u (x(t),uj(t), vq, X(t)) = 0. (And, if 
the linearization along Z) is output controllable, v§ = 1.) 

Two special cases merit separate statements. The first is the one when no 
terminal constraints are imposed. Then the above equations reduce to: 

x = H\(x, u>, 1, A)' , x(a) = x° 

A = -n x (x,u,l,X)', A(r) = (p.(i(r)))' 

and TL u (x(t) ,u)(t) , 1, A(i)) = 0. (Linearized output controllability holds by defi- 
nition, if r = 0.) 

The other special case is the one when the final state is specified to be a 
given x = x ! (i.e., K(x) = x — x ! ). Then, any optimal control Zj for which 
the linearization along uj is controllable must be such that there is a solution 
(*(■), A(-)) of 

x = H\(x, w, 1, A)' , x(a) = x° , x(t) = x 
A = -H x (x,w,l, X)' 

and, again, H u {{x(t), u>(t), 1, X(t)) = 0. (Linearized output controllability con- 
dition holds; cf. Corollary 9.2.9.) 

Systems Affine in Controls, Quadratic Costs 

An interesting class of systems consists of those that are affine in controls, that 
is, 

f(x,u) = F{x) + G{x)u 

for some n- vector function F and some n x m matrix function G. (This is as in 
Equations (4.18), (4.22), and (5.10), but we write here F instead of go! note that 
linear systems have this form, with F a linear function and G constant.) For 
such systems, it is natural to consider costs along trajectories that are quadratic 
in u: 

q(x,u) = u'R(x)u + Q(x) 

for some in x m symmetric matrix function R(x) and a scalar function Q(x). 
Because of the general assumptions made on / and q, we have that F, G, R, 
and Q are (at least) continuously differentiable. In this context, we will say that 
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q is regular if it is quadratic in u and R(x) is nonsingular for each x € X, and, 
in that case, we introduce the map 

M : Ixl"^R": (a;, 77) h- &(&, p(ar, r?))' + f x (x, p(x, v))'v 

where p(x,rf) := —i2R{x)^ 1 G{x)' , q. (For example, if R and G are independent 
of a; then M(x,if) = —Q*(x) — F*(x)r].) 

Proposition 9.2.10 Assume that u> G £u(cr, r) is optimal for a; , with respect 
to the system i = _F(a;) + G(a;)u, where g is quadratic in u and regular, and that 
the linearization along Z) is output controllable. Then, there are a vector v d W 
and an absolutely continuous function A : [it, t] — » R™, such that (x, A) = (a;, A) 
is a solution of the following system of In differential equations: 

x = F(x) - -G(x)R(x)- 1 G(x)'X, x{a) = x" (9.16) 

\=-M(x,\), A(r) = ( P 4x(t)) + v'K*(x(t)))' . (9.17) 

Moreover, if F, G, R, and Q are of class C k , fc = 1, 2, . . . , oo, or (real-)analytic, 
then cD must be of class C k , or analytic, respectively* 

Proof. As H u (x, u,l,r]) = 2u' R{x) + rj'G(x) for all vectors u, x, rj, the equation 
TL U = can be solved for u, to give u = —(l/2)R(x)~ 1 G(x) / r). Therefore, 

C(t) = -^^(^-^(xWVAW. (9.18) 

This expression for lu may then be substituted into the differential equations 
for x and A, giving the two-point boundary-value problem in Equations (9.16) 
and (9.17). The function M has one less degree of differentiability than the data 
defining the problem. Thus, the set of 2n differential equations has a right-hand 
side of class C k " 1 on x and A. It follows that (x, A) is of class C k in t (and 
is analytic, in the analytic case); cf. Theorem 1 (p. 57). Thus, from Equation 
(9.18), we obtain the same differentiability for lo. I 

Remark 9.2.11 Recall that the proof of kth order differentiability in the C k 
case is very easy: suppose that z is an absolutely continuous function that solves 
z = W(z). Assume that z has been shown, by induction, to be of class C in t. 
Then, provided that W is of class C k , W(z(t)) is of class C k in t, from which 
it follows that z is of class C k , from which we conclude that z is in fact of class 
C k+1 . □ 

The next two problems show that Theorem 43 (p. 403) provides the same 
characterizations as, respectively, the Linear-Quadratic problem treated in Sec- 
tion 8.2, and the least-squares problem covered in Theorem 5 (p. 109), when 
costs and constraints are appropriately defined. 



"That is to say, u> is almost everywhere equal to a function that has those properties. 
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Exercise 9.2.12 Consider the optimal control problem for a linear time-invar- 
iant system x = Ax + Bu with quadratic costs, that is, q(x,u) = u' Ru + x'Qx 
and p(x) = x'Sx, where R, S, Q are symmetric matrices, Q and S are both 
positive semidefinite, and R is positive definite. We assume that there are no 
final-state constraints, K = 0. Let P(-) be the solution of the matrix Riccati 
differential equation P = PBR~ l B'P - PA - A' P - Q with final condition 
P(t) = S. Consider the solution (x(-),A(-)) of the two-point boundary value 
problem obtained in Theorem 43 (p. 403). Show that A(i) = 2P{t)x(i) for all 
t £ [a, t\. {Hint: You may want to make use of Lemmas 8.2.8 and 8.2.9 — with 
these, the proof takes only a few lines.) □ 

Exercise 9.2.13 Consider the optimal control problem for a linear time-invar- 
iant controllable system x = Ax + Bu with quadratic cost q(x,u) = \\u\\ and 
p(x) = 0. Assume now that, in addition to the initial state a; , a final state x l is 
also specified. Write X(t) = $(r, t)'v, for some vector v £ R™, and solve 7i u = 
for ui(t) in terms of X(t). Next substitute this expression into the differential 
equation for x and use the form of the solution to solve for v in terms of x° 
and x . At this point, you will have an explicit expression for uj. Compare with 
Theorem 5 (p. 109). □ 

Example 9.2.14 Consider the system with X = K 3 , IX = K 2 , and equations: 

X\ = U\ 

x 2 = u 2 
± 3 = x x u 2 ■ 

(This is an example of a completely controllable system for which no contin- 
uous feedback stabilization is possible, and is a small simplification of Exam- 
ple 5.9.16; see also Exercises 4.3.14 and 4.3.16. The unsimplified equations 
could be used too.) Consider the problem of minimizing the control energy 
J{u) = J g ll w ( s ))ll ds, with initial and final states fixed [K[x) = x — x^). We 
claim that every optimal control must be of the form 

»w - . (_•£&.'!)) <™> 

for some a, 6, a £ K. (We will not address the converse question of when such a 
control is optimal.) To prove this, take any optimal control u(-). If u = 0, then 
U is of the form (9.19), with a = and b, a arbitrary. (Observe that the control 
u = is, obviously, the unique optimal control in the interval [a, t] that takes 
x° to itself.) So we assume from now on that u(-) = (wi(-) ■U2(-))' is optimal 
and u ^ 0. 

We first show that the linearization along the corresponding trajectory x(-) = 
</>(•, a, x Q , u) is controllable. We have, along this trajectory, the linearized system 
matrices: 

/ o o\ 

A(t) = and B{t) = 

\u 2 (t) 0/ 




1 














1 











Xl(t) 


u 2 {t) 


«i(t) 



9.3. Excursion into the Calculus of Variations 409 

Observe that Bit) happens to be absolutely continuous, in this example, because 
x\ is. Let Bq := B and B\ := ABq — Bq; then, 



(-Bo , Bi 



This matrix cannot have rank < 3 for almost all t £ [a, r], because u ^ 0. There- 
fore, the linearized system described by (A(t), B(t)) is controllable, by Corol- 
lary 3.5.18. (The statement of that result requires that A and B be smooth, but 
smoothness is not required when only first derivatives are evaluated. The argu- 
ment in that case is in fact easy enough that it is worth giving here: if the lin- 
earized system would not be controllable, then there would be some vector p^ 
such that p'$(r, t)B(t) = for almost all t G [a, r]. Thus, since $(r, t)B(t) is ab- 
solutely continuous, also -j- t p'<S>(T,t)B(t) =p'${T,t)A(t)B{t)-p'${T,t)B(t) = 
0, from which it would follow that q(t)'(B (t), B^t)) = for q(t) = $(r, t)'p =£ 0, 
so (Bo(t),Bi(t)) would have rank < 3 for almost all t.) 

Since we are dealing with the constraint K(x) = x — x l and the linearization 
is controllable, we have linearized output controllability (arguing as in Corol- 
lary 9.2.9), and we are in the situation of Proposition 9.2.10. Thus we know 
that u(t) is analytic in t, and we have that, for some solution A of A = —A(t)'X, 

Ui(t) = -Ai(t) and u 2 {t) = -X 2 {t)-X 3 {t)x 1 {t). 

Moreover, A2 and A3 are constant; we let Aa(t) = a. Using that Xi(t) = —au 2 {t), 
we have then that 

u\{t) = au 2 (t) and u 2 {t) = — crai(t) . 

Therefore u must have the form (9.19). □ 

9.3 Excursion into the Calculus of Variations 

Let X be an open subset of 1™. By a curve we will mean a Lipschitz continuous 
map x : [a, r] — > X. Let 

q : IxR"^R 

be a given C 2 function. As a Lipschitz continuous x is absolutely continuous 
and has an essentially bounded derivative x, the integral J q(x, x) dt is well- 
defined for any curve. Suppose also given an interval [o~, r] and a point x° G X. 
The following is the classical problem in the calculus of variations: 

minimize the integral J q[x, x) dt over all curves x : [a, t] — > X with x(o~) = x° . 

This is the "endpoint-unconstrained" problem. In addition, one often considers 
also the problem in which a terminal constraint 



x(t) = X 
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where x is prespecified, is imposed; we will call this the "endpoint-constrained" 
problem. 

Example 9.3.1 Consider the problem of finding the (Lipschitz continuous) 
y : [0,1] — » M>o that joins the points (0,1) and (l,j/i) and has the prop- 
erty that, when its graph is revolved around the x-axis, there results a sur- 
face of minimal area. From elementary calculus, the surface in question has 
area A = J Q 2iry-\/l + (dy/dx) 2 dx. Thus, the minimization problem is the 
endpoint-constrained problem for which cr = 0, t = 1, x° = 1, X — yi, and 
q(x,x) := x\/\ + x 2 (the constant factor 2n is irrelevant). □ 

We call a curve x : \(t,t] — > X, with x(a) = x , optimal if J q(w,x)dt < 
J q(x, x) dt for every x : [a, b] — » X with x(o~) = £. For the endpoint-constrained 
problem, we ask that x(t) = x i and that J q(ui, x) dt < J q(x, x) dt for every 
x : [a, r] — * X with x{t) = x ! . 

Both versions (x l free, or specified,) are (very) special cases of the optimal 
control problem treated earlier. Indeed, we consider the system x = u, evolv- 
ing on the state space X and with IX = K m . Note that there is a one-to-one 
correspondence between curves and controls, since x = u (and an absolutely 
continuous function has an essentially bounded derivative if and only if it is 
Lipschitz). We introduce the same cost q, thought of as a function q(x, u), and 
p = 0. For the endpoint-constrained problem, we let K(x) = x — x ! . Admissi- 
bility of a control u) amounts to asking that the solution x u remains in X, and 
a control uj is optimal for x a in the sense of optimal control if and only if the 
corresponding trajectory x is optimal in the variational sense. 

From Theorem 43 (p. 403), then, we may conclude as follows. 

Corollary 9.3.2 If a curve x is optimal for the endpoint-unconstrained prob- 
lem, then q u (x(t), x(t)) is absolutely continuous, 

-q u (x(t),x(t)) = Qx(x(t),x{t)) ( 9 - 20 ) 

for almost all t € [a, r] , and 

q u {x{T),x{r)) = 0. (9.21) 

If the curve x is optimal for the endpoint-constrained problem, then the function 
q u {x{t) ,x(t)) is absolutely continuous, and Equation (9.20) holds for almost all 
t. 

Proof. Take first the unconstrained case (K = 0, so we can take vq = 1). 
Theorem 43 (p. 403) says that there is an absolutely continuous solution of 
A = (—q x )' (note that A(t) = 0), with boundary condition A(r) = 0, such that 
H u (x(t),x(t), 1, X(t)) = 0. Here we have Ti(x, u, I, if) := q(x, u) + r/'u, so TL U = 
means that A = {—q u )' along x. Therefore, (d/dt)(q u ) = —A' = q x , as claimed, 
and the boundary condition A(t) = gives q u = at the endpoint. 
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Now consider the endpoint-constrained problem, that is, K(x) = x — x l . 
The linearization of x = u along any trajectory is again x = u and is therefore 
controllable, so that Corollary 9.2.9 applies. The same equation then results. I 

The differential equation (9.20) 

d_ 

dt qu ~ qx ' 

satisfied along optimal curves, is called the Euler or Euler-Lagrange equation 
associated to the variational problem. If x is of class C 2 , then it solves the 
equivalent second-order differential equation: 

q U uX + q-uxi - q x = . (9.22) 

Remark 9.3.3 Observe that in the endpoint-unconstrained case one has the 
boundary conditions x(a) = x° and (q u )\ t=T = 0. In the endpoint-constrained 
case, instead, the boundary conditions are x{a) = x° and x(t) = x l . Thus 
the "right" number of boundary conditions are imposed, and one may expect, 
under appropriate technical assumptions, to have only a discrete number of 
solutions. The Euler-Lagrange equation is merely a necessary condition that 
must be satisfied by an optimal curve. If an optimal curve is known to exist, 
and if the Euler-Lagrange equation (and boundary conditions) lead to a unique 
solution, then this solution must be the optimal one. We do not treat here 
existence questions; there is a vast literature on the subject. □ 

Example 9.3.4 Consider the problem of minimizing L x 2 + x 2 dt, x(0) = 0, 
cc(l) = 1. That is, q(x,u) = u 2 + x 2 . We have q u = 2u, q x = 2x. So the 
Euler-Lagrange equation, which must be satisfied for any possible minimizing 
x, is dx/dt = x. This means that x must be absolutely continuous and (a.e.) 
x = x. So x(t) = ae* + 6e~*, for some constants a, b. The boundary conditions 
give that x(t) = . \ . sinht. □ 

& ^ ' sinh 1 

Exercise 9.3.5 For each of the following problems, solve the Euler-Lagrange 
equation (with appropriate boundary conditions). 

1. H x 2 - x 2 dt, x{0) = x{ir) = 0. 

2. Jq Ax 2 + 2xx - x 2 dt, x(0) = 2, cc(tt) = 0. 

3. J Q cos xdt, x(0) = 0. (Find only solutions x(-) that are continuously 
differentiable.) 

4. L Vl + x 2 dt, x(0) = a, x(l) = b, where a and b are any two constants. 

In the last problem, after you have solved it, explain why the solution should 
have been obvious. □ 
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Note that, in Example 9.3.4, the equation x = x tells us that x must be 
absolutely continuous, so a third derivative exists almost everywhere, and, with 
an inductive argument, we obtain that all derivatives must exist. Of course, 
differentiability is also clear once the solution in terms of sinh is obtained, and 
this solution makes it clear that x is even a real-analytic function of time. This 
is a fairly general fact, which we discuss next. 

We restrict our attention, for simplicity of exposition, to the one-dimensional 
case, n = 1. 

The variational problem is said to be regular if 

Quu{x,u) > for all (:e,u)gXxR. 

For instance, in Example 9.3.4, q(x,u) = u 2 + x 2 , so q uu = 2, and regularity 
holds. 

Lemma 9.3.6 Suppose that the problem is regular and that q G C k , for some 
k G {2, ..., oo}. Then every solution of the Euler-Lagrange equation, and hence 
every optimal curve, is (at least) of class C on t € (<j, r). If q is real-analytic, 
then every such solution is analytic as well. 

Proof. Consider the following map: 

a : IxR-tlxB : { x ,u) t— > (x,q u (x,u)) . 
At each point, the Jacobian of a is 

1 

Qux Quu 

which has full rank = 2. Therefore, for each two pairs (xo,Uo) 6 1x1 and 
(xq,Pq) 6lxl such that a(xQ,u ) = (cco,.Po) there is some mapping 7 : D — » 
1x1, defined on some open neighborhood T> of (xo,Po)i with the properties 
that 7(2:0, po) = {xo,Uoj and 

a{j{x,p)) = (x,p) 

for all (x,p) G T> (Inverse Function Theorem). Moreover, 7 has the same degree 
of regularity as q u , so 7 is of class C fc_1 (and is analytic if q is analytic). If we 
write 7 = (71, 72), the map 72 : T> — > R is so that 

q u (%,i2(x,p)) = p 

for each (x,p) G T>. Furthermore, since for each fixed x G X, the map u 1— * 
q u (x,u) is one-to-one (because its derivative is everywhere positive), we have 
that, for each (x,p) G 2?, u = 72(3?, p) is the only solution in R (not merely in a 
neighborhood of Uo) of the equation q u (x, u) = p. 

Now let x = x be any solution of the Euler-Lagrange equation. Pick any 
to G (o-, t). Write 

p{t) = q u (x(t),x(t)) (9.23) 
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for all t, and let Xq := x(to), uq := x(to), and po ■= p(to)- Let 7 be a function 
as above, denned on some neighborhood V of (x(to),p(to)). Since both x and p 
are (absolutely) continuous (in the case of p, this is asserted in Corollary 9.3.2), 
there is some neighborhood X of to so that (x(t),p(t)) € T> whenever t € X. 
Therefore, for all t € X, we can solve Equation (9.23) for x(t) using 7 2 : 

x(t) = l2 (x(t),p(t)). (9.24) 

So we have that (x(t),p(t)) is a solution, on the interval X, of the system of 
differential equations 

x = 72 (x, p) 

v = qx{x,j2{x,pj), 

(the second one is the Euler-Lagrange equation). Thus, (x,p) has one more 
degree of regularity than 72 and q u , namely k continuous derivatives (and is 
analytic, in the analytic case); cf. Propositions C.3.11 and C.3.12. I 

Corollary 9.3.7 For a regular problem, every solution of the Euler-Lagrange 
equation, and hence every optimal curve, satisfies 

q(x(t),x(t)) - x(t)q u (x(t),x(t)) = c (9.25) 

for some constant c. 

Proof. We simply compute 4z (q — xq u ) = —x (q U u x + Qux x ~ Qx), an d remark 
that this vanishes, because of Equation (9.22). I 

Remark 9.3.8 We introduced two notions of regularity, one for scalar varia- 
tional problems and another one for optimal control problems with quadratic 
costs (cf. Proposition 9.2.10). It is possible to provide a common generalization 
for both results, in terms of the Hessian with respect to u of the cost function 
q and a one-to-one condition on its gradient. However, the two cases that we 
presented are the ones most commonly used, and are the easiest to check. □ 

Example 9.3.9 We now study the Euler-Lagrange equation for the problem 
of minimal surfaces of revolution (cf. Example 9.3.1). That is, we consider the 
problem, on X = R>o, 

1 

xy / l + x 2 dt , aj(0) = 1 , x{l) = y x . 
'0 

Let a; be a solution of the Euler-Lagrange equation. Since 

x 

(l + u 2 ) 3/2 
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is real-analytic and positive, we have a regular problem, and x(-) is real-analytic. 
Equation (9.25) gives 

x = cyl + x 2 , 

for some constant c which, because x(0) = 1, must satisfy < c < 1. From 



• 2 i ' x 



x l = ( -\ - 1 (9.26) 

it follows that 

1 
XX = —xx . 

c 2 

We will assume that it is not the case that x = (which can only happen if 

x = c, and thus y\ = 1). So, by analyticity, there is some nonempty open 

interval where 

1 

Cr 

and hence there are constants ai,/3i so that 

x(t) = u\ cosh (t/c) + j3\ sinh (t/c) 

on that interval. By the principle of analytic continuation, x must have this 
form on the entire interval. The initial condition x(0) = 1 implies that a\ = 1, 
and (9.26) evaluated at t = gives (/?i/c) 2 = (1/c) 2 — 1 and hence (3\ = 
±\/l-c 2 G (-1, 1). Pick d e E so that 

tanh d= (3\ , 

which implies that coshrf = 1/c. Then (using cosh(;z + y) = cosh z cosh y + 
sinh x sinh y ), 

, s cosh (t cosh d + d) 

X ®= cosh. ■ ^ 

Every minimizing curve must be of this form. 

We must, however, meet the second boundary condition: x(l) = y\. This 
can be done if and only if one can solve 

cosh (cosh d + d) 
coshrf 

for d, which requires y\ to be sufficiently large (approximately > 0.587); in gen- 
eral, there may be none, one, or two solutions. (For instance, if y\ = coshl, 
the solutions are d = and d « —2.3. The integrals L x\J\ + x 2 dt are, re- 
spectively, approximately 1.407 and 1.764. So if a minimizer exists, it must be 
x(t) = coshi. It turns out, but this is harder to prove, that this function is in- 
deed the unique global minimizer. Incidentally, the truncated cone obtained for 
the straight line x(t) = 1 — (1 — cosh \)t, would give an integral of approximately 
1.447.) 
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Establishing minimality is nontrivial, but is well-studied in the literature. 
For small j/i, or more generally, for minimization on a large interval (instead of 
[0, 1]), it is physically intuitive that the true minimal-area surface will consist 
of two disks (at the endpoints), connected by a line. This is not representable 
as a graph of a function, which means that there are no minimizing extremals 
of the type we have considered. (In control-theoretic terms, the derivatives are 
"impulses" and one is led into the area of "impulsive controls" .) □ 

9.4 Gradient-Based Numerical Methods 

We now present some remarks concerning numerical approaches to the mini- 
mization of J(u>). In order to concentrate on the basic ideas, we assume that 
there are no endpoint constraints (no K) and also that q = (though this last 
simplification can easily be overcome by a system extension, as done in the proof 
of Theorem 43). Thus J{oS) = p(x u (t)), and we assume that p : X — > K is C . 
We still assume that IX is open and / is differentiable. 

Classical iterative techniques start with a control uj\ : [cr, r] — » It, and 
attempt to improve recursively on this initial control, obtaining a sequence 
u>i,u>2, ... so that J{lo\) > J(u)<i) > . . .. Of course, just as with any numerical 
procedure of this type, it may well be impossible to improve after a certain u>k, 
and, even if improvements are always possible, there is no guarantee that the 
sequence J(lou) will converge to the infimum value of J . Here, we concentrate 
on the individual recursion step. (In practice, reinitialization is used in order to 
restart from another uj\ an iteration which fails to decrease fast enough.) 

Suppose, then, that at a certain stage we have a candidate control uj € 
£u(c, t), which is admissible for the initial state x°. We attempt to compute a 
perturbation /i G C^(a, r) with the property that 

J(Zj + hfi) < J(u) (9.28) 

for a sufficiently small "step size" h > 0. Let 

P : (-/} ,/j )-»K n : h>-> <f>{T,<T,x°,uj + hn) (9.29) 

be the map considered in Corollary 9.1.1. Then 

J(u + hn)-J(w) = p(0(h)) - p(P(0)) = (p°P)*(0)h + o(h), 

will be negative for all small enough h > 0, provided that [i is chosen such that 
(p o /3)*(0) < 0. By the chain rule and Equation (9.7), this last derivative is: 

P*(x(t)) / &(T,s)B(s)tJ,(s)ds, (9.30) 

where A, B, and $ are as in Corollary 9.1.1. Thus the objective has been 
reduced to: 
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find an input fj, such that (9.30) is negative. 

If this goal can be met, then, for all small h > 0, uj + h\x will be an admissible 
control in C.u{<^, t) for the same initial state x°, and thus we will have obtained 
a new control which leads to a smaller cost. The process can then be repeated, 
after redefining uj := uj + hfj,. (Note that, however, in practice, once that such a 
/i has been found, one first performs a "line search" over h, a scalar optimization 
over the parameter h, in order to minimize J{uj + h/j,), instead of simply taking 
any small enough h.) 

Of course, it may be impossible to make (9.30) negative, for instance because 
P„(x(t)) = 0. So we will assume that 

p*(5?(r)) + 0. 

This is often a reasonable assumption: for instance, if p(x) = \\x\\ then this 
property will only fail if x = 0, but then, the minimum has already been found. 
Furthermore, we assume from now on that the linearization along the trajectory 
(x, uj) is controllable as a time-varying linear system, that is, that the reacha- 
bility map for this linearized system, 

L : jZ™(a,T)^R n :/*•—»•/ $(r, s) B(s) n{s) ds 

•J a 

is onto. Equivalently (cf. Theorem 5 (p. 109)), 

W{o-,t) = / $(r,s) B(s) B(s)' $(t,s)' ds 

Jo 

is a positive definite matrix. When this happens, we say that the input uj is 
nonsingular for the state x . 

Because of controllability, there exist inputs \i such that 

Lu. = - P *(x(t))' (9.31) 

and any such input helps meet the goal of making (9.30) negative (namely, 
the value becomes — ||p*(a;(r))|| ). There are infinitely many solutions fi to 
Equation (9.31). One natural choice is the least squares solution, that is, the 
unique solution of minimum L 2 norm, 

H ■= -L*p4x(t)Y (9.32) 

where L# denotes the pseudoinverse operator discussed in Theorem 5 (p. 109). 
This gives the formula 



H(t) = -B{t)' $(T,t)' •W{<j,t)- 1 P ,(x(t)Y . (9.33) 

-\\x\ 
solving p(x) 



When p(x) = \\x\\ , this choice of /! amounts to a Newton-method iteration for 
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Another natural choice for the perturbation \x is: 

/i := -L*p*(x(T))' (9.34) 

where L* is the adjoint of L. This gives the formula 

H{t) = -B(t)' $(T,t)' P *(x(t))' (9.35) 

which has the advantage that no matrix inversion is required. With this choice, 
we have that L/x = — LL*p*(x(t))' = — W(a, t)p*(x(t))' . Thus, the expression 
in (9.30) becomes 

P*(x(t)) f $(T,s)B( S )v(s)ds = -p*(x(t))W(<t,t)p*(x(t))', 

J (7 

which is negative (because W(a, r) is positive definite), so indeed we made a 
good choice of perturbation fi. This is, basically, a steepest descent method for 
minimizing J . 

Exercise 9.4.1 The matrix W(a, r) = j T <&(r, s)B(s)B(s)'§(t, s)' ds appears 
in the computation of the perturbation given by formula (9.33). It would seem 
that in order to compute W(a, r), one must first solve the differential equation 
x = f{x,ui), then solve for the fundamental solution $ of the variational equa- 
tion, and, finally, integrate numerically using the formula for W(a, r). Such an 
approach involves large amounts of storage (values of x to be used in the fun- 
damental equation, as well as in B(t), values of $, etc). There is an alternative 
way of computing W(a, r), in "one pass", however, as follows. Let 

A(x,u) = f x (x,u) and B(x,u) = f u (x, u) , 

seen as functions from X x IX into K nx ™ and R" xm respectively. Now consider 
the following system of n + n 2 differential equations for x : [a, r] — > X and 
Q : [<t,t] ^R" x ": 

x = f{x,w) 

Q = A(x,co)Q + QA(x,tu)' + B(x,oj)B(x,u>)' . 

Show that the solution of this equation with initial conditions x(a) = x° and 
Q(a) = is so that Q{t) = W(a, r). □ 

Exercise 9.4.2 Consider the system with X = K 3 , II = K 2 , discussed in Ex- 
ample 9.2.14. Take the control lu € £u(0, 27r) whose coordinates are 

Ui(t) = , v,2{t) = sin(t) . 

Since w ^ 0, it is nonsingular for any initial state x°. 

(a) Compute a formula for the pseudoinverse operator L#, when the control ui 
is used, and x° is arbitrary. 

(b) Provide a formula for the basic recursive "Newton" step in (9.33), when 
p(x) = \\x\\ , and compute two iterates, starting from any nonzero state of your 
choice. (You may use h = 1.) □ 
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9.5 Constrained Controls: Minimum Principle 

Theorem 43 (p. 403) deals exclusively with problems in which the optimal con- 
trol takes values in the interior of the constraint set. When the input- value set 
is not open, however, it may provide no useful information. 

As an illustration, consider the problem of minimizing x(l), for the sys- 
tem x = u with U = [—1,1], when the initial state is x(0) = 0. Obviously, 
the optimum is attained with ui = — 1, a control which takes boundary val- 
ues. If we apply the Theorem, we obtain, letting Ti,(x,u,l,r]) = r/u (note 
that we may take Vq = 1, since there are no constraints on final states), that 
Ti u (x(t),u}(t), 1, A(t)) = X(t) = 1 (since A = and A(l) = 1); thus it is impossi- 
ble to satisfy Equation (9.14). Of course, the Theorem could not be applied to 
begin with, since U is not open in this example. However, this example already 
serves to indicate a possible alternative. The condition 7i u = is necessary for 
7i to have a local extremum with respect to u. If we look at 7i itself, not its 
derivative, we have that, along trajectories, H(x(t),u>(t), 1, A(t)) = Lu(t). Thus 
the true solution lu = —1 is obtained if, instead of setting derivatives to zero, 
we ask that 7i(x(t),u, 1, A(t)) should be minimal, among all u € U. This is 
what the Minimum (or "Maximum" , depending on sign conventions) Principle 
asserts. 

In this section, we do not need to assume that / : X x 11 — > R n is continuously 
differentiable nor that the cost functions are differentiable in u. We assume only 
that / : X x IX — > X is C 1 in x, and that /, as well as its Jacobian matrix of 
partial derivatives f x , are continuous on x, u (here 11 is a metric space and X an 
open subset of some Euclidean space). Similarly, the three functions 

q:XxU^R , p:X^R, K : X -> W 

are assumed to be C 1 on x and continuous on u. The Hamiltonian is defined 
just as before, 

H : X x It x R x R n -> R : (x, u, ry , rj) i-> r] q(x, u) + r]'f(x, u) . 

We only prove a result in the (much) simpler case in which the final state is free 
(no final state constraints given by U K" ) , in which case 7/0 will evaluate to 1 , so 
one can also write Ti = q(x, u) + rf f(x, u). 

Theorem 44 (Minimum Principle,) Assume that Zj G £u(c, t) is optimal 
for x° . Then, the solution A : [<j, r] — > R n of the final-value problem 

\(t) = -q x (x(t)Mt)y ~ A(ty\(t), \(r) = (p.(2(r)))' (9.36) 

is so that 

H(x(t),w(t),l,\(t)) = mmH(x(t),u,l,\(t)) (9.37) 

uEU 

for almost all t G [a, r] . 



9.5. Constrained Controls: Minimum Principle 419 

Proof. Suppose that lu is optimal, and let x(-), A(-), and $ be as in the proof 
of Theorem 43. Just as in that proof, we first establish the case q = and then 
derive the general case from it by considering an extended system. 

Pick any instant to G (c, r) such that x(to) exists and x(to) = f(x(to),u>(to))', 
the set of such £ ' s h as a complement of zero measure. We will show that 
Equation (9.37) holds at to- Take any element u G IX; we must show that 

W(s(to),w(to),l,A(t )) < H{x{t ),u,l,\{t j). (9.38) 

For each e > small enough, we may consider the control lu 6 defined as 
follows: 

u>(t) if t & [t - e, t ] 
u if t G [to — e,to] 

(this is often called a "needle variation" of lu around t = to). The control lu £ 
is admissible for x° provided that £ is small enough. Indeed, by continuity of 
4>, there are two neighborhoods Co and O of x{to), and some £o > 0, with the 
following properties: (a) if x G O then the restriction of lu to [£o,t] is admissible 
for x, and (b) if x G Co then the constant control lu = u on any interval [r, r+£o] 
of length £o is admissible for x, and (/>(£, r, x, lu) G O for every t G [r, r + eg]. We 
may, of course, pick O to have compact closure, so that f(x,u) and f{x,ui(t)) 
are bounded whenever x G O and £ G [<7, r] . 

Now take any £ < £o such that 5;(io — s) € Co- It follows that one may solve 
with constant control lu = u starting from x{to — e) at time t = to — £, and at 
time t = to the resulting state is in O, so the "tail" of ui is admissible after that 
instant. The complete trajectory obtained by concatenating these solutions is 
then a solution on the entire interval [a, r] . 

We denote x £ := x ule , that is, x e (t) = </>(t, <7, X , u> e ) for all t G [c,r]. Next 
we compare x e (to) and x(io)- We have 

[•to 

x e (t )-x(to - e) = / f(x e (s),u)ds 
•>t a -e 

rto rto 

[f{x e {s) 7 u) - f(x{t ),u)]ds + / f{x{t ) 7 u)ds 

lt -£ Jt -e 

= 5i(e) + ef{x(t ),u) 

for some function <5i which is o(e) as e\0. (Such a 5i can be obtained as follows. 
Let M be an upper bound on the values of ||/(cc,w)|| and ||/(x,o)(t))|| for x G O. 
Then, for each s G [t — e,to], \\x e (s) — x(t n — e)\\ = \\x s (s) — x s (t — e)\\ < eM, 
and also ||x(io) — x(t — e)\\ < eM, so ||x e (s) — x(to)|| < IsM. Now let c be a 
Lipschitz constant for f(-,u) on O; it follows that we have a bound as above 
with S{e) := 2ce 2 M.) 

On the other hand, because we took a point to where the differential equation 
holds (actually, only differentiability from the left is required), we have that 

x(t ) - x(t - e) = ef(x(to),Lu(t )) + 5 2 (e) 
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where 82 is also o(e) as e\0. Thus 

x e (to) - x(to) = e [f{x{t ),u) - f{x{t ),X{t ))] + 8(e) (9.39) 

where 8 = 81 — 62 is o(e). 

Consider the map : x 1— > </>(r, to, X, v), where v is the control lu restricted 
to the interval [io>Tl; © is defined on a neighborhood of Xq := a?(io), and hence 
is defined for x e (to) when e is small enough. The first part of Corollary 9.1.1, 
applied on the interval [£oj t ]j with respect to the input v, gives that 

6(ar) = 6(4) + Q*(x° )(x-x° ) + g(\\x - x° Q \\) 
= 9(4) + ®(T,to)(x-x° ) + g(\\x - x° \\) , 

for some function g which is o(r) as r — > 0. We substitute x = x s (to) into this 
expression, and use Equation (9.39) as well as the equalities 0(x e (£ o )) = X e (r) 
and 6(a;g) = x(t), to obtain as follows: 

x £ (t) = x(t) + e*(r,t ) [/(»(*o),«) - /(«(*o),w(t ))] + o(e) ■ 
Finally, since p is differentiable, we may also write 

p(x e (t)) = p{x(t)) + p*(x(r)) (x s (t) -x(t)) + o(e) 

= p(x( T )) + ep*(x(T))$( T ,t ) [f(x(t ),u) - f(x{t )Mto))] + o(e). 

Since, by optimality, p(x(r)) = J(u>) < J{u) e ) = p(x £ (t)), we have that 

ep.(s(r))*(r,t ) [/(s(*o),«) - /(s(to),w(*o))] + o(e) > 

for all e > sufficiently small, so taking e\0 we obtain that 

p.(x(r))fc(r,to) [/(»(*o),«)-/(s(*o),w(to))] > 0. 

Let A be the solution of \{t) = -A{t)'\{t) with final value A(t) = (p*(x(r)))'. 
We have proved that 

A(io)'[/(^o),w)-/(x(io),w(i ))] > 0. 

Recall that 7i(x,v, 1,rj) = rj'f(x,v) for all v € 11. Thus inequality (9.38) indeed 
holds. 

Now the case of arbitrary q is obtained exactly as in the proof of Theorem 43. 
There are no changes in the proof until the last sentence, which is now replaced 
by the equality (with the obvious notations): 

H*{x*{t),u,l,\*{t)) = H(x(t),u,l,X{t)) 

for all t; the fact that u>(t) minimizes TC&(x&(t),u, 1, A#(i)) over u G U gives 
the desired conclusion. I 
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9.6 Notes and Comments 

The material on optimization and the Calculus of Variations is classical; see, 
for instance, [48], [134], [151], [189], [247], [248], [402], [442]. Path minimization 
problems were posed and solved at least since classical times (Dido's problem, to 
find a curve of a given length which encloses the largest possible area, appears 
in Virgil's writings). However, the origins of the field may be traced to the 
challenge issued "to all mathematicians," in 1696, by Johann Bernoulli, to solve 
the brachystochrone problem: "If in a vertical plane two points A and B are 
given, then it is required to specify the orbit AMB of the movable point M, along 
which it, starting from A, and under the influence of its own weight, arrives at B 
in the shortest possible time." This problem, which was soon afterwards solved 
by Newton, Leibniz, and many other mathematicians of the time (including 
Johann and his brother Jakob), gave rise to the systematic study of the Calculus 
of Variations. The article [393] explains the historical development, taking the 
more general point of view afforded by optimal control theory. 

A good reference for the material in Section 9.4 is [78]. Under mild condi- 
tions, it is possible to prove that generic controls are nonsingular for every state 



x 



o. 



see [373] and [374], and the related papers [99] and [100]. 



Section 9.5 merely skims the surface of the subject of the Minimum (or 
Maximum) Principle. In particular, we did not provide a treatment of the more 
general case in which final-state constraints K(x) = are present. However, the 
form of the result is easy to guess, and we state it precisely here; see [134] for a 
proof. Suppose that K : X — » K r is also of class C . Assume that uj G Cu{v, r) 
is optimal for a; . Then, there exist a scalar uq € {0, 1} and a vector v GW , not 
both zero, so that the solution A : [er, r] — » 1" of the final- value problem A(t) = 
-v q x (x(t)Mt))' -A(t)'\(t) with A(r) = (v P*(x(t))+v'K*(x(t)))' (cf. (9.13)) 
is so that 7i{x(t), ui(t), uq, A(t)) = rava ul ziil-L{x{t),u,i/Q 1 \(t)) for almost all t G 
[a,r]. 

It is also possible to give even more general results, for instance, to deal 
with mixed initial/final state constraints, as well as non-fixed terminal time and 
time-optimal problems. For the latter, one must consider variations in which 
a piece of the control is deleted: such shorter controls cannot be optimal, and 
the directions generated in this manner lead to an appropriate Hamiltonian. 
(The only time-optimal problem considered in this text, cf. Chapter 10, is for 
linear systems, and can be developed in a simple and self-contained fashion 
using elementary techniques from convex analysis.) 

Several extensions of the Maximum Principle have been developed during the 
past few years. These provide "high order" tests for optimality, and in addition 
permit the study of far more general classes of systems, including those in which 
the dynamics does not depend in a Lipschitz continuous manner (or is even 
discontinuous) on states. A promising direction, in [390], develops an approach 
to generalized differentials ( "multidifferentials" ) , and proposes their use as a 
basis for a general nonsmooth version of the maximum principle; references to 
the extensive literature on the subject are also found there. 



422 9. Optimality: Multipliers 



Chapter 10 

Optimality: Minimum-Time 
for Linear Systems 

We consider time-invariant continuous-time linear systems 

x = Ax + Bu (10.1) 

with the control- value set II being a compact convex subset of R m . As usual, 
a control is a measurable map u> : [0,T] — > R m so that ui{t) € IX for almost all 
t £ [0,T]. We denote by ££(0,T) the set consisting of measurable essentially 
bounded maps from [0, T] into R m (when m = 1, just £°°(0, T)) and view the set 
of all controls as a subset Cu(0, T) C £^(0, T). In this chapter, we write simply 
£u instead of £^, because, II being compact, all maps into U are essentially 
bounded. 

For each fixed x° e R n and T > 0, 

a x ^ T : >C(0,T)^R™ : lu ^ e TA x° + f e {T - s)A Buj{s) ds 

Jo 

is the end-point map or reachability map that sends an input u> into the state 
reached from x° at time T if this input is applied. That is, a x o ,t{ x °) = X (T), 
where x(-) is the solution of the initial value problem x(t) = Ax(t) + Buj(t), 
x(0) = x°. We denote the reachable set in time exactly T from state x° by 

n T (x°) = a x o, T (Ax(0,T)) 

and let 1Z(x°) := Ut>o T^ T { x °)- For any two fixed x° and x f G lZ(x°), we wish 
to study this problem: 

Find a control which takes state x° to state x in minimal time. 

The study of time-optimal problems for linear systems can be pursued as 
a simple application of basic facts from convex analysis, and we do so here. 

423 
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We restrict attention to the time-invariant case merely in order to make the 
presentation as clear as possible. However, it is not difficult to generalize the 
material here, with similar proofs, to time- varying linear systems. 

10.1 Existence Results 

Existence of optimal controls will be easy once we establish the following fact. 

Theorem 45 For each x° £ R" and T > 0, the set lZ T (x°) is compact and 
convex. 

Since a x o T is an affine map, and £u(0, T) is convex (because 11 is), it follows 
that lZ T (x°) is convex. The compactness part of Theorem 45 can be proved by 
first defining a topology that makes £u(0, T) compact and then using that a x a T 
is continuous when this topology is used for £u(0,T). Since the continuous 
image of a compact set is compact, this will show compactness of 1Z T (x°). We 
will follow this idea, but, in order to make the presentation as elementary as 
possible, develop everything in terms of convergence of sequences. 

Weak Convergence 

Definition 10.1.1 The sequence {u>k} of elements in £^(0,T) is said to con- 
verge weakly to uj £ C^(0,T), denoted LUk^Lu, if: 

T ,-T 

(p(s)'wk(s) ds — > / cp(s)'u)(s) ds as k — » oo , 
'o Jo 

for each integrable function tp : [0,T] — > M. m J □ 

We write 

IMI^ := sup || W (t)|| 
te[o,T] 

(where ||w(t)|| means Euclidean norm in M m , and the supremum is interpreted 
as essential supremum); this is the infimum of the numbers r\ with the property 
that ||w(t)|| < rj for almost all t £ [0,T]. When we say that a sequence {tOk} is 
bounded, we mean that there is some rj so that HwfcH^ < f] for all k. We prove 
later the following result. 

Lemma 10.1.2 Every bounded sequence {uj^} in £^(0,T) has some weakly 
convergent subsequence. 

Lemma 10.1.3 Assume that lo £ £^(0,T) and the sequence {iOk} in Cu(0,T) 
are such that, for each interval J C [0,T], 



Wfe(s) ds — > / uj(s)ds as k — > oo . 
J J J 



Then, w e£ u (0,T). 



t Prime indicates transpose. 
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Proof. Let L be the intersection of the sets of Lebesgue points in (0, T) for the 

I f* 

h Jt 



coordinates of u). Then j- J ||w(s) — oj(t)\\ ds — > as h\0, for each t G L, so 



1 j-t+h 

lim - / uj(s)ds = w(t) VteL. (10.2) 

h\fi h J t 

Pick any t G L. We will show that u>(t) G IX. Since the complement [0, T] \ L 
has measure zero, then uj(t) € 11 for a. a. t. 

Choose h > 0. Applying the hypothesis to the interval [t, t + h], and dividing 
by the constant h, we have that 

1 j-t+h -I j-t+h 

Qk,h ■= t I uj k {s)ds -> - / w(s)ds 

as fc — > oo . If we show that 

qk,h€U, (10.3) 

then this limit shows (because li is closed) that 

- / w(s)ds e U. 

As this holds for any h, taking limit as h\0 and using Equation (10.2), gives 
u>(t) G 11, as wanted. 

We now prove (10.3). Since II is a closed convex set, we can represent II as 
an intersection of closed half-spaces, U = C\ reR {v G M m | A r v < p r }, where R 
is some (in general, infinite) index set. Pick any r G R. Then 

/ i j-t+h \ i jt+h i r t+h 



K'Qk,h = V I t / w fe (s)ds| = - / X r 'uj k (s)ds < - / p r ds = p r . 
Therefore g^/j G II. I 

Corollary 10.1.4 If w fc ^cj and uj k G £ U (0,T) for all k, then w G C u (0,T). 

Proof. It is enough to show that the hypothesis of Lemma 10.1.3 holds, that 
is, that 



u>k,i(s) ds — > / uji(s) ds (10.4) 

for each interval J and for each coordinate i = 1, . . . , m. Fix any such J and 
i. Consider the function </?(s) = Ij(s)ei, where Ij is the characteristic function 
of J and ej is the zth canonical basis vector in K m . Since L tpi j(s)'uj k ds = 
j j^>k,i{s)ds, and similarly for ui, the convergence (10.4) holds. I 

Together with Lemma 10.1.2, we will then have: 
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Proposition 10.1.5 Every sequence {iOk} of elements in Cu(0,T) has a sub- 
sequence which is weakly convergent to some u> € £u(0,T). □ 

In order to prove Lemma 10.1.2, we first provide a simple characterization 
of weak convergence of bounded sequences. 

Lemma 10.1.6 Let {lu^.} be a bounded sequence in C°°(0,T), and let u> € 
jC°°(0,T). Suppose that, for each interval J C [0,T], 



Wfe(s) (is 



w(s) rfs as fc — > oo . 



Then, LUk —> ui. 



Proof. Let u> and {tOk} be as in the statement. Pick a number rj so that 
ll^felloo 5= ?7 f° r &U fe- Applying Lemma 10.1.3 with IX = [ — 77, 77] C R, we have 
that also ||c<j || 00 < r\. We claim that 



Uk{s) ds — > I u)(s)ds 
s J s 

for every measurable subset S C [0, T]. To see this, pick any e > 0, and for this 
£ pick a finite set of disjoint intervals I\ , . . . , Ig so that the set Q = ii |J . . . |J It 
satisfies that the symmetric difference QAS has measure 



meas (QAS) < 



3ry 



(One can always find such intervals: first cover S by an open set Qq so that 
meas (Qa\S) < e/2; since Q is a union of countably many disjoint open intervals, 
it has a subset Q as above so that meas (Qq \Q) < e/2.) Pick now a fcp so that 



k > k 



u>k(s) ds — u>(s) ds 



<3T j = 1 > 



J. 



Then, for each k > fco, Lwt(s) ds — J„oj(s) ds\ is bounded by 



uJk(s) ds — / LOk(s) ds 

s 



u)(s) ds — I u)(s) ds 
Q Js 

l 



£ 



Uk(s) ds — I u>(s) ds 



which in turn bounded by V'i~~^ r lj~~^^M = £ - As e was arbitrary, J„ u>k(s) ds 
J s u)(s) ds. 

Next, we claim that 



T pT 

<p(s)ujk(s) ds — > / ip(s)uj(s) ds 
Jo 



(10.5) 
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for every simple function tp = 5Zf=i c i^Si- Indeed, 

,-T p ,■ p ,. ,.t 

I tp(s)u>k{s) ds = / c i uJk(s) ds — > \ Ci u)(s) ds = / <p(s)u>(s) ds . 

Jo i=1 J Si i=1 J Si Jo 

Finally, we claim that Equation (10.5) holds for every integrable function 
tp : [0,T] — » R. To see this, pick any e > 0; there is then a simple func- 
tion 9 so that J Q \9(s) — ip(s)\ds < e/(Arj). (First find a bounded tp that ap- 
proximates <p in C , and then approximate tp uniformly by a simple function.) 



Pick fco so that, for all k > fco, 
/o <p(s)u k (s) -/„ ^(s)w(s)rfs 



/„ (6»(s)w fc (s) - 0(s)w(s)) ds < e/2. Then 



is bounded by 



(tp(s) - 9(s))u>k(s) ds 



(tp(s) - 9(s))u{s) ds 



(9(s)iv k {s) - 9{s)lu{s)) ds 



< 2r] / \6{s) - ip{s)\ ds + 
Jo 



(9(s)uj k (s) - 6(s)u>(s)ds) 



< e 



for all such k. I 

Example 10.1.7 Let T > 0, and pick any number p G [0,1]. Define uj k : 
[0,T] — > R, A; = 1, 2, . . ., by partitioning [0,T] into fc equal intervals, and, in 
each such interval, making its average be p: 



LO k (t) 



l ifte [»f,(t + p) 

otherwise . 



0,. 



,fc- 1 



We claim that 

w 
UJk^ P 

(function constantly equal to p) . If Jq is an interval of the special form 



T T 



with i € {0, . . . , k — 1}, then J, uj(s)ds = pT/k. So, more generally, if Jo = 
[i-jr, (i +p)]f]j with i + P also in {0, . . . , k — 1}, J^ ui(s)ds = ppT/k = Jj pds. 
For any interval J, there is some interval Jo C J of the form Rti(* + p)t] 
so that J \ Jo has measure less than IT jk. Therefore, f, u) k (s)ds — /, prfs < 
T , |wfc(s) — p| <is — > 0. The claim then follows from Lemma 10.1.6. □ 
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Corollary 10.1.8 Assume that u> G £~(0,T), and that the sequence {w/t} in 
£u(0,T), are such that, for each interval J C [0,T], 



LUk(s) ds — > / uj(s)ds as k — > oo . 

Then, cj G £u(0, T) and W& -^> u>. 

Proof. The first conclusion was given in Lemma 10.1.3. The hypothesis means 
that J jOJk.i(s) ds — » JjU>i(s)ds for every interval J and each coordinate i = 
1, . . . , 77i. Furthermore, since IX is bounded, the coordinate sequences {uJk.i] are 
bounded. Pick any tp G £^j(0,T), so each of its coordinates ^j[0,T] — > M. is 
integrable. Then 



I tp{s)' io k {s) ds = Y^ / 
Jo i=1 Jo 



T 

(Pi(s)'u!k,i(s)ds 



converges to 



m „t ,-T 

V" / cpi(s)'u>i(s)ds = / ^(s)'w(s) 
i=1 Jo Jo 



<is , 



by Lemma 10.1.6. 



Exercise 10.1.9 Let V C II be so that the convex hull co (V) = 11. Show that 
for each lu G £tx(0,T) there is some sequence i>k^>u> so that fk{t) G V for all 
t G [0,T]. (Suggestion: you may want to argue as follows. First, show that 
lu can be weakly approximated by piecewise constant controls, i.e., of the form 
^2a n it e Iji u i' f° r intervals J, C [0, T] and elements m G IX. Next argue that, on 
each interval J, a constant control with value u = $^ i=1 Pi^j, with X^P* = 1j 
Uj G V, pi > for all «, can be in turn weakly approximated by controls with 
values in V. For this last approximation, you may think first of the special case 
r = 2: in that case, the sequence 0Jk( v i ~ v 2) + v 2 converges to pv\ + (1 — p)v 2 , 
if UJk is the sequence constructed in Example 10.1.7.) □ 

Proof of Lemma 10.1.2 

Assume that H^fcH^ < r\ for all k. Define fk(t) ■= J LUk(s)ds. Each element of 
the sequence {fk} is Lipschitz continuous with bound rj\ 

\\Mh) - Vk{t 2 )\\ < f ||«fc(*)|| ds < ij|ti-t 2 | , (10.6) 

Jti 

and bounded by r/T. Therefore the sequence {vk} is equibounded and equicon- 
tinuous, and the Arzela-Ascoli Theorem can be applied, from which one con- 
cludes that it has a convergent subsequence. Without loss of generality, we 
assume that i>% — ► v as k — > oo, for some continuous v : [0,T] — > K m . From 
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Equation (10.6), taking limits as k — > oo, we have that v is also Lipschitz 
continuous with bound r\. Thus v is absolutely continuous, so there exists an 
integrable function qj : [0, T] — > K m , namely u> = dv/dt, so that v(t) = L uj(s)ds 
for all t. Since u> is the (a.e.) derivative of v, and v is Lipschitz with constant 
rj, it follows that la is almost everywhere bounded by r\. Thus lu G £^(0,T), 
and we are only left to establish that uj k —> u>. Because of Corollary 10.1.8, it is 
enough to show that 

r>b pb 

/ uj k (s)ds — > / uj(s)ds 

•J a J a 

for all subintervals [a, b] of [0, T]. Take any such a < b. As 

6 
uj k (s)ds = v k (b) - v k {a) 

and v k — y v pointwise (even uniformly), we have that J u)k{s)ds — > v(b) — v{a). 

Since u> = dv/dt, the last expression is the same as J oj(s)ds, so the proof is 
completed. I 

Proof of Theorem 45 

Convexity is clear, as remarked earlier, because a x o t ls an affine map, and 
£u(0,T) is convex. 

We now prove that 1Z T (x°) is compact. Let {x|} be a sequence of elements 
of H, T (x°), and let the controls uj k € £u(0,T) be so that a x o T (u>k) = x k 
for each k. Picking a subsequence, we assume without loss of generality that 
ujk^>LO for some control u> € £u(0,T). Let x ! := a x o_ T (uj) G 1Z T (x°). We claim 
that x k —y x f , which will prove that the original sequence has a convergent 
subsequence. Indeed, the ith coordinate of x k , i = 1, . . . , n, has the form 

T 

ipi(s)'u) k (s)ds, 

where (pi(t) is the ith row of e f - T ~ t ' B. By weak convergence, 

/ ipi(sYuj k (s)ds — » / <pi(s)'u)(s)ds 
Jo Jo 

as k —y oo, and this last is the ith coordinate of x ! . I 

Remark 10.1.10 The reachable sets 1Z T (x°) can be proved to be closed even 
for a fairly wide class of nonlinear systems. Consider systems of the type 
x = f{x) + X^i u i9( x ) (that is, affine in controls), where /, g\ , . . . , g m are 
locally Lipschitz vector fields. We use a x o T (uj) to denote the solution x(T) 
of the differential equation, when the input is u = to G C\i(Q,T) and the ini- 
tial condition is x(0) = x° , provided that the solution is defined on the entire 
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interval [a, r]. Thus, a^ is a map T> x o T — > R", where T> x a T is a subset of 
£-u{&)T) (an open subset in the supremum norm, cf. Theorem 1 (p. 57). As- 
sume that T> x a T = £ix(cr, t). Then the set lZ T (x°) is compact. Indeed, the 
same proof works as in the linear case; the only technical fact needed is that 
a x a T (u) k ) — » a x a ^t{ uj ) whenever uj k ^uj, and this follows from Theorem 1. 

Observe also that Exercise 10.1.9 allows us to conclude, for any V C IX for 
which co (V) = U, that K T {x°) = clos (K^{x )) for all x a and T, where K$(x°) 
denotes the set of states reachable from x° using controls with values in V. □ 

Exercise 10.1.11 Consider the nonlinear, affine in controls, system x = x 2 + u, 
with 11 = [—1, 1] (any other compact convex set 11 could be used). Show that 
there are states x° and times T for which 1Z {x°) is not compact. Why does 
this not contradict the discussion in Remark 10.1.10? □ 

Existence of Time-Optimal Controls 

Theorem 46 If x G !Z(x°) then there is a control which takes state x° to state 
x l in minimal time. 

Proof. Let T be the infimum of the set of nonnegative real numbers 

{t\x { eK\x a )}. 

We need to show that x G 1Z (x°). Assume that t k \T are such that x G 
1Z tk (x°) for all k. For each k, let u>}~ be a control steering x° to x l in time t}~. 

Let x k be the state reached from x° by restricting u k to the interval [0,T]. 
Each x k € 1Z T (x°), and the sequence {x ! k } converges to x l . Indeed, we have 

x l -x{ = (e £kA -l)x{+ f " e^ £k -^ A Buj k {T+s)ds 7 

Jo 

where we are denoting e k := t k — T, so 

ll^f rr* II <* \\& e kA r|| ||^v,f || _i_ n ^ 

\\ — k\\ — — /c ' k ^ 

where c is an upper bound on the possible values of e tj4 i?u , for t G [0,t] and 
U G It, and r is any number majorizing t k for all k. On the other hand, 

x [ = e tkA x s + / e (tk - s)A Bio k ( S )ds, 
Jo 

implies 

\\x{\\ < e r||A " ||a; f || + ct , 

so the sequence ||a;||| is bounded. Since ||e £fc — /|| — > as k — > oo, we have 
that x k — > x i . 

Since R T (x°) is closed (by the previous lemma), it follows that x ! G 1Z T (x°), 
as desired. I 

We call a control that takes x° to x f in minimal time a time-optimal control. 
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Remark 10.1.12 The assumption that IX is compact cannot be dropped from 
Theorem 46 (p. 430). To see this, consider the one-dimensional system x = u. 
If 11 = [0, 1) then the infimum time for reaching x f = 1 from x° = is T = 1. 
However, 1 ^(O) (because 1 = L cu(s)ds implies L (1 — ut(s))ds = 0, but 
1 — lu(s) > for all s, so uj(s) = 1 for a. a. s, a contradiction). Thus boundedness 
by itself is not enough. Neither is being closed, as illustrated with the same 
system and II = R, since, for the same x l and x° , one has T = but 1 is not in 
ft°(0) = {0}. □ 

10.2 Maximum Principle for Time-Optimality 

We review first an elementary fact from convex analysis. Let C C 1" be a 
convex set and let x ! G R™. The hyperplane 

H = {x | rfx = 8} 

(where rj G R™ is nonzero) is said to separate x and C if rj'z < 8 for all z G C 
and r]'x i > 8. In particular, r/ [z — x l ) < for all z € C. If there is some such 
H, then x l cannot be in the interior of C: otherwise, there is some e > so 
that z := x l + er\ G C, and hence 

»/(*-a; f ) = e|M| 2 >0, 

a contradiction. Conversely, assume that x is not in the interior of a convex set 
C. Then there exists a hyperplane separating x and C; this is a standard result, 
see for instance [330], Theorem III. 11. 6. or [188], Theorem 1.9.1 and Corollary 
1.9.2. (When x l G' C, this result is a consequence of the Hahn-Banach Theorem, 
and is valid for arbitrary locally convex topological vector spaces instead of 
merely for subsets of R"; see e.g. [348], Theorem II. 9. 2.) We summarize for 
future reference: 

Lemma 10.2.1 Let C C R™ be a convex set and x l G R™. Then, there exists 
a hyperplane separating x l and C if and only if x l int C. □ 

Assume now that H = {x | r/'x = 6} separates x l and C and that x belongs 
to C . Any such H is said to be a support hyperplane to C at x f . Since x f G C, 
the inequality rfx 1 < 8 holds, so r)'x i = 9. In other words, 7J is a support 
hyperplane to C at ar if and only if there is some nonzero vector r\ G R" so that 
H = {x | rf {x — x l ) = 0} and 

rf (z-x*) < VzG C. (10.7) 

Any such r\ G R" is said to be normal to C at x l . So, from the above Lemma, 
we have the following fact, which holds in particular for the sets C = 1Z T (x°), 
for every T > and x° G R™. 
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Lemma 10.2.2 Let C C W 1 be a closed convex set and x l s C. Then cc f e dC 
if and only if there is a normal to C at i f . □ 

After these preliminaries, we return to the time-optimal problem. 

Lemma 10.2.3 Assume that x° and x l are two states, and T > is such that, 
for some sequence t^yT, 

x { £K th {x°) (10.8) 

for all k. Then x { g mtU T {x°). 

Proof. Consider any fixed £&. Since the point x l does not belong to the convex 
set TZ tk (x°), in particular it is not in its interior, so there is some hyperplane 
{x | f] n 'x = 8 n } separating x and 1Z tk (x°). Thus, for each k, 

?7 fe '(z-a; f ) < for all zeTZ tk {x°). (10.9) 

As one can multiply each rj^ by a positive scalar, we may, and will, assume 
without loss of generality that r\k has unit norm. By compactness of the unit 
ball in R", a subsequence of {rjk}, which we will take to simplify notations 
to be again the same sequence, converges to some (unit) vector r\. Pick any 
Z € 1Z (x°). Let Zk — > z with Zk € lZ tk (x°) (for instance, truncate to the 
intervals [0, £&] any given control that steers x° to z in time T, and argue that 
Zk — > z as in the proof of Theorem 46 (p. 430)). Then Equation (10.9) gives 
that r]k' \Zk ~ x ) < for all such k. Taking limits as k — > oo, Equation (10.7), 
with C = lZ T (x°), results, so by Lemma 10.2.2 the conclusion follows. I 

States reached with time-optimal controls are in the boundary of the respec- 
tive reachable sets: 

Corollary 10.2.4 If x ! € lZ T (x°) and T is minimal with this property, then 
x { e dTZ T {x°). 

Proof. If T = then x f = x° and lZ°(x a ) = {x }, and the result is clear. So 
take T > 0. Minimality means that property (10.8) holds, for some (in fact, 
any) sequence t n /T. Lemma 10.2.3 then implies that x l intlZ T (x°); since by 
assumption x ! G lZ T (x°), the conclusion follows. I 

A (very) partial converse to the above result, valid for trajectories starting 
at x° = 0, is as follows. The system (10.1), with control-value set 11, satisfies 
the small-time local controllability (STLC) condition (at x° = 0) if 

0eintft £ (0) Ve>0. 

It was shown in the proof of Proposition 3.6.3 that a sufficient condition for 
STLC is that the following two properties hold: 

• the matrix pair (A, B) is controllable; 



10.2. Maximum Principle for Time-Optimality 433 

• G int It. 

Lemma 10.2.5 Assume that the STCL holds and < S < T. Then K s (T) C 
int7e T (0). 

Proof. Pick any x { = J^ e^ s ~ s "> A Buj{s) ds G 1Z S (T). Let V be an open subset 
of TZ T ~ s (0) which contains (STLC assumption). Consider the open set 

Q ■= { e SA v + x i | v€V }. 

Since e is invertible and V is open, this is an open set as well, and x l G Q 
(obtained when v = 0). We claim that Q C H T (0). Indeed, pick any v G V and 
consider the state 



,SA„ 



w + a; f = e Sj4 i;+ / e {s ~ s)A B(v(s)ds. 



This is the state reached from v in time S, when applying control to, so z G 
1Z (v). Since v G 7^ T_S (0), transitivity of reachability gives that z G 7?. T (0), as 
claimed. I 

This Lemma has the following obvious corollary. 

Proposition 10.2.6 Assume that x f G dlZ T (0) and that the STLC is satisfied. 
Then T is minimal with the property that x l G 1Z T (0). □ 

Remark 10.2.7 For nonzero initial states a; , it may be the case that x l G 
dlZ T (x°) yet T is not minimal for steering x° to x l , even if the STLC holds. 
For example, consider the system x = x + u with 11 = [—1,1]. Here ^(l) = 
[1, 2e — 1], so 1 G dH (1), but the minimal time for going from 1 to 1 is T = 0, 
not T = 1 . And even for initial state x° = the result may fail, if the STLC does 
not hold. For an example, take x = u, U = [1,2], and note that 7?. 1 (0) = [1,2], 
so 1 G dH (0), but 1 can be reached in time 1/2 from 0. □ 

We have the following characterization of boundary points of reachable sets. 

Theorem 47 (Maximum Principle for boundary of reachable sets.) Assume 
that x G dlZ T (x°), and that uj is any control steering x° to x in time T . Then, 
for every normal vector r\ to lZ T (x°) at x l , the (necessarily everywhere nonzero) 
solution A of the adjoint linear differential equation 

X = -A'X, A(T) =rj (10.10) 

satisfies 

X(t)'Bu(t) = maxX{t)'Bu (10.11) 

for almost all t G [0,T]. 

Conversely, suppose that ui is any control steering x° to x l in time T . As- 
sume that there is some nonzero solution X of Equation (10.10) so that Equa- 
tion (10.11) holds for almost all t. Then, x l G dlZ T (x°). 
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Proof. Let r\ be normal to 7^ T (a; ) at x l , i.e., 

V' {z ~ x f ) < for all z G TZ T {x°) . (10.12) 

So 

rf f e^ T -^ A B{cj{ S )-^{s))ds < (10.13) 

Jo 

for all controls to G >Cu(0,7 1 ). That is, it must hold that 

i-T 

X(s)' 'B (uj(s) - ui(s)) ds < (10.14) 

for all U), where A(s) := e( T ~ s ' r\ is the solution of Equation (10.10). 

Let J C [0, T] be the set of points where Equation (10.11) fails to hold. Pick 
any subset 1/oClI that is dense and countable. From the definition of J, the 
continuity of X(t) Bu on u, and the density of Uq, it follows that for each t G J 
there is some Ut € Uq so that 

\{t)'Bu{t) < \{t)'Bu t . 

We claim that J has measure zero. Otherwise, by countability of Uq, there must 
exist some fixed element «o G Uq, and some subset Jo of J with nonzero measure, 
so that X(t) Bui(t) < \{i) Bu$ for all t € Jo- Let uj(t) be the control which 
equals uq whenever t G Jo and equals ui(t) elsewhere. Then the integrand in 
Equation (10.14) is everywhere nonnegative, and is positive on Jo, contradicting 
the inequality. 

We now prove the converse statement. Let Zj steer x° to x in time T, and let 
A be a nonzero solution of Equation (10.10) so that Equation (10.11) holds for 
almost all t. Consider any other control u) G Cu{Q,T). From Equation (10.11), 
we know that 

\{t)'Bu{t) > \{t)'Buj{t) 

for almost all t € [0,T]. Therefore, 

rf f e ( - T -^ A B{uj{s)-Z{s))ds < 0. 
Jo 

As this is true for any u>, Equation (10.12) holds, that is, r\ is a normal vector 
to TZ T {x°) at x f . I 

Corollary 10.2.4 asserts that, if T is the optimal time for reaching x f from 
x°, then x f G dlZ T (x°). Thus, we have the following necessary condition for 
optimality as a Corollary of Theorem 47. 

Theorem 48 (Maximum Principle for time-optimal control of linear systems.) 
Assume that uJ is a control steering x° to x l in minimal time T > 0. Then, 
x l £ dlZ T (x a ), and for every normal vector r/ to lZ T (x a ) at x f , the (necessarily 
everywhere nonzero) solution A of the adjoint linear differential equation (10.10) 
satisfies Equation (10.11) for almost all t G [0,T]. □ 
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Definition 10.2.8 The input lo G £u(ct, t) satisfies the maximum princi- 
ple (for the time-optimal control problem) if there is some nonzero solution X 
of Equation (10.10) so that Equation (10.11) holds for almost all t. □ 

Another terminology often used is "u> is an extremal" . Observe that the 
property means that there exists some nonzero vector n G R" so that 

t)'e (T " f)A Bu(() = for almost all t . 

An equivalent statement is that there must exist some nonzero vector 7 G R" 
so that r y'e~ tA Buj(t) = for almost all t (just use 7 = e TA 77). 

We have proved that a time-optimal control must satisfy the maximum prin- 
ciple, and that a control which satisfies the maximum principle must lead to 
boundary points. Under additional assumptions, such a control is also time- 
optimal, thus having a partial converse to Theorem 48, as follows. 

Proposition 10.2.9 Assume that the STLC holds and that either x a = or 
x l = 0. Then, if 10 steers x° to x l in time T, while satisfying the maximum 
principle, then u> steers x° to x { in minimal time T . 

Proof. The case x° = follows from the previous discussion. This is because 
Theorem 47 implies that x ! G dlZ T (0), and, if the STCL holds, this in turn 
implies optimality (cf. Proposition 10.2.6). 

We will reduce the case x f = to the case x° = 0, as follows. First we 
introduce (A, B) := (—A, —B), and show that the STLC (with the same control- 
value set IX) holds for (A,B). Next, we remark that, if u> satisfies the maximum 
principle and steers x Q into 0, then the new control v G £u(0,T) given by 

v(t) := ui{T-t) (10.15) 

satisfies the maximum principle with respect to the system x = Ax + Bu and 
steers into x° for that system. Applying the previously established case, we 
conclude that T is the minimal time for steering x° to 0, for the new system, 
and this is equivalent to T being minimal for the original problem. We now 
fill-in the simple details. 

To see that the STLC holds for (A, B), fix any T > and let V be an 
open neighborhood of included in 1Z T ({)). It will be enough to show that 
—e~ T V C 1Z T (Q) (reachable set for new system), since — e~ T V is an open set 
containing 0. So pick any x° = —e~ T x l , with x l G V. As x l G 1Z T (0), there is 
a control u G £u(0,T) so that x l = L e^ T ~ s > Boj(s)ds. Therefore 

x° = [ e sX Bu{s)ds = [ e^ T ^ X Bu{r)dr G 7e T (0) , 
./o Jo 

where we changed variables by r = T — s and defined v as in Equation (10.15). 

Suppose that u) satisfies the maximum principle, that is, there is some 

nonzero vector 7 so that 7'e - * Bu>(t) = max u6 u 7' ' e~ tA Bu for all t € J, where 
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J is a set so that meas ([0,T] \ J) = 0. Let v be as in Equation (10.15). Take 
the nonzero vector 

7 := -e TA '~i = -e~ TA '~f. 

Then for each s = T — t G {T — t,t G J}, the maximum principle holds for v 
with respect to the new system: 

7 'e- sX Bu{s) = ie {T - s)A Buj{T-s) = j'e- tA Buj(t) 

= max~f'e~ tA Bu = ma,xj'e^ T ~ s ^ A Bu = ma,x : y'e~ sA Bu . 

ueu ueu ueu 

Finally, we remark that if u> is any control steering a state x° to a state x l , 
then v defined as in Equation (10.15) steers x l to x° for the new system (and 
vice versa). This is clear either from the explicit formulas for solution, or simply 
by noticing that if x = Ax + Buj then z(t) := x(T — i) satisfies z = Az + Bv. 

Applied to lu G L\i satisfying the maximum principle and steering x° to 0, we 
conclude that v steers to a; for the new system, and is therefore (case where 
initial state is zero) time-optimal. But then T is minimal so that G lZ T (x°), 
since if = a x o S (LU ) with some S < T then vo{t) = l<Jq(S — i) contradicts 
optimality of v. I 



10.3 Applications of the Maximum Principle 

The maximum principle leads us to study, for each nonzero vector 7, the fol- 
lowing function: 

M 7 : R -> E m : t -> B' ' e~ tA '7 . 

This is an analytic function of t, and its derivatives at t = are all zero if and 
only if i A l B = for % = 0, 1, . . ., so: 

Lemma 10.3.1 The matrix pair (A, B) is controllable if and only if M 7 ^ 
for each 7 7^ 0. □ 

Thus, if (A, B) is controllable, and 7 / 0, M 7 (t) can be zero only for a 
discrete set of t's. 

The "bang-bang" principle says that time-optimal controls take the most 
advantage of possible control actions at each instant. The name is motivated by 
the particular case of an interval U = [u, u] , where optimal controls must switch 
between the minimal and maximal values u and u. There are various theorems 
that make this principle rigorous. The following is the simplest one. 

Theorem 49 (Weak bang-bang.) Assume that the matrix pair (A, B) is con- 
trollable. Let uj be a control steering x° to x l in minimal time T > 0. Then, 
to € dll for almost all t G [0, T]. 
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Proof. For each r\ £ R" and each t £ K, we introduce the following function: 

H Vtt ■ U-> E : mh M v (t)'u. 

Thus n n j{u) = rj'e~ Bu for each w. 

Note that by the maximum principle, Theorem 48 (p. 434), x ! £ dlZ T (x°). 
Fix any normal vector 77 to TZ (x ) at x . Then the solution of Equation (10.10), 
A(t) := e( T_t ) A 77 = e~* A 7, where 7 := e Tj4 77 7^ 0, is so that Equation (10.11) 
is satisfied (a.e.). That is 

H yt {u}{t)) = max/J 7jt (u) (10.16) 

for almost all t £ [0,T]. In general, let q € M m and consider the function 
/ : U — ► R, f(u) = q'u. If / is not identically zero, then every point in U at 
which / achieves a maximum relative to U must belong to the boundary of IX 
(if u € intll, then u + eq £ II for some small e > 0, but then /(u + eg) = 
f(u) + e \\q\\ > f(u)). In particular, by Lemma 10.3.1, /i 7] t can be the zero 
function only for t in a finite set J C [0,T]. So w(i) G dli for each t £ J for 
which Equation (10.16) holds. I 

Exercise 10.3.2 Give an example of a pair (A, B) (necessarily, not control- 
lable), a convex compact II, and x°, x f , T, and uj, so that oj steers a; to x f in 
minimal time T > but ui(t) G' dli for any i G [0, T]. □ 

For a large class of systems, one can obtain uniqueness. 

Definition 10.3.3 The system x = Ax + Bu with control-value set 11 is nor- 
mal* if, for each nonzero vector p, for almost allt G R the function /j, Pt t achieves 
its maximum at exactly one point of XL. □ 

In the normal case, given any nonzero vector r\ £ R n and any T > 0, we 
define the control 

uj v {t) = argmax ueU yu 7it (u), (10.17) 

where 7 := e r\. The "argmax" in Equation (10.17) is well-defined for almost 
all t, and we let ui n (t) be any fixed element of U if t is in the set (of measure 
zero) where the maximum is not uniquely attained. 
The main result for the normal case is as follows. 

Theorem 50 Assume that the system (10.1) with control-value setXi is normal. 
Then, for each two states x° and x l £ lZ(x°), there is a unique time-optimal 
control Zj steering x° to x i . 



*The term "normal" is used in a different manner than for "normal vectors", but, unfor- 
tunately, this is by now standard practice. 
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Proof. Let T be minimal with x l G lZ T (x°). We fix a normal vector r] to 
R T (x°) at x , and define uj v as in Equation (10.17). We claim that this is the 
unique optimal control. Indeed, assume that ui is such that 0^0^(0)) = x . Then 
the maximum principle and normality (cf. Equations (10.16) and (10.17)) give 
uj(t) = U) v (t) for almost all t G [0, T]. I 

Two important special cases of normality are as follows. 

A strictly convex set C is a convex set with the property that every sup- 
porting hyperplane H to C intersects C at exactly one point. For instance, 
any ball in Euclidean space is strictly convex. An equivalent statement for 
compact convex C C R l is that C is strictly convex iff each nonzero linear func- 
tion / : K ( — » K achieves its maximum on C at a unique point of C (since 
H = {x G R™ I f(x) = max ce c'/( c )} is a supporting hyperplane to C, and, 
conversely, any supporting hyperplane can be represented in this fashion) . 

We say that 11 is a hypercube if there are real numbers u^ <Ui, i = 1, . . . , m, 
so that 

U = [u 1: ui] x [u 2 ,u 2 ] x ... x [u m ,u m ] 

Observe that for any function of the form f(u) = p'u so that p G K m has all 
coordinates pi 7^ 0, then / achieves its maximum on U at the unique point 
(«i, . . . ,u m )', where 

'* l P * >{ i (10.18) 

u, lfp, <0. v ' 

Lemma 10.3.4 Assume that either: 

1. the pair (A, B) is controllable and 11 is strictly convex, or 

2. each pair (A, hi) (pi is the ith column of B) is controllable and U is a 
hypercube. 

Then the system (10.1), with control-value set IX, is normal. 

Proof. By definition of strict convexity, /i 7jt achieves its maximum at exactly 
one point whenever /i 7it is nonzero; by controllability, this happens for almost 
all t. In the hypercube case, we need to guarantee that the coordinates of 
p = M~(t) = B'e~ f 7 are nonzero; but these coordinates are b' i e~ tA 7, and are 
not identically zero because each (A, bi) is controllable, so the unique minimizing 
point is given as in Equation (10.18). I 

It is worth summarizing the conclusions in the hypercube case. Note that 
these conclusions apply to all single-input (m = 1) systems, since every closed 
convex subset of M. is of the form [u, u] . 

Theorem 51 Suppose that each pair (A, bi) is controllable and that IX is a hy- 
percube. Then, for each two states x° and x l G lZ(x°), there is a unique time- 
optimal control oj steering x° to x l . Moreover, there is a nonzero vector 7 G W 1 
so that the ith coordinate of uj, i = 1, . . . ,m, satisfies 

~n\ / «< if7'e~ ty % >0 nma\ 

U ^ = { u t if 7 ' e -^<0 (1 °- 19) 
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for almost all t € [0,T]. □ 

Given the control u> taking x° to x l in optimal time, we can always modify 
lu on a set of measure zero, so that Equation (10.19) holds for all t in the set 

J = {t\ ie~ tA bi ± for alU = 1, . . . ,m} . 

The complement of this set is the union of the sets of zeros of the analytic 
functions 7'e - ' 6; on the interval [0,T], so is finite. So we can rephrase our 
conclusion by saying that, for the normal hypercube case, optimal controls are 
piecewise constant, taking values in the vertices of U, and have a finite number 
of possible switches, in the complement of J. 

The following is an interesting remark concerning the geometry of reachable 
sets. 



Proposition 10.3.5 Assume that the system (10.1), with control-value set U 

is 
convex 



normal. Then, for each state a; and each T > 0, the set lZ T (x°) is strictly 



Proof. Suppose that H = {x | n'x = 8} is a supporting hyperplane to 1Z T (x°). 
Let u> n be as in Equation (10.17). Pick any two points x\ and X2 in H f)lZ T (x°); 
we must prove that x\ = x 2 . Choose two controls u>i so that Xi = a^o t(wj), 
i = 1,2. Then, because X\ and x 2 both belong to dlZ T (x°), and r\ is normal to 
lZ T (x°) at both x\ and x 2 , Theorem 47 (p. 433) says that 

u>i(t) = uj v = L0 2 {t) , 

so x\ = a x a T {uJi) = a x a T (ijj2) = x 2 , as desired. I 

Scalar Systems 

We specialize now to single input systems (m = 1), and write b instead of 
B = b\. In general 11 = \u, u], but we will take, in order to simplify the 
exposition, u = —1 and u = 1. We assume that the pair (A, b) is controllable. 

For each two states x° and x { , there is a unique time-optimal control Zj 
steering x° to x f , and there is a nonzero vector 7 € R" so that 

u(t) = sign( 7 'e-* A &) (10.20) 

for all t qL Sj_t, where 

S* 7]T = {te [0,T] \ 1 'e~ tA b=Q} 

is a finite set. 

Although finite, the sets 5-^t may have arbitrary cardinalities. For instance, 
if we take the harmonic oscillator matrix 

1 
-1 
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we have 



-tA / \ /r> -. \ I cos t — sin t\ ( 



(OD^^lJ =( 01 )Uin^ cos. J (J J = COSi ' 

Exercise 10.3.6 Give an example of a controllable system with IX = [—1,1], 
having the following property: for each integer k there exists a state x k such 
that the (unique) minimum-time control steering to x k switches sign k times. 
(Hint: To prove optimality, use Proposition 10.2.9.) □ 

However, there is a case of some interest where a very strong conclusion may 
be drawn. 

Proposition 10.3.7 Suppose that the matrix A has only real eigenvalues. 
Then, for each 7, b, and T, S~t nas a t most n — \ elements. 

Proof. By an exponential polynomial we will mean a function of the form 

k 

P(t) = "£Pi(t)e ait , 

i=l 

where ct\ < a-i < . . . < ctk are real numbers and each pi a polynomial. We write 
7r(P) := Xa=i(l + ^»)) where 8i is the degree of pi. We will prove that any such 
P, if not identically zero, can have at most 7r(P) — 1 (distinct) real zeros. This 
will give the result, because, using the Jordan form of A, we know that 7'e - * b 
is an exponential polynomial with 7r(P) < n. 

We proceed by induction on n(P). If n(P) = 1 then P = ce at has no zeros. 
Suppose now that the result has been proved for n. Take P with n(P) = n + 1 
and not identically zero. Assume that P would have > n + 1 zeros. Then 



P (t) := e~ a ^P(t) = Pl (t) + £>i(t)e< Q '- ai 



)* 



has the same > n+ 1 zeros. This implies that Pq = p^ + . . . has at least n zeros. 
But 7r(Pg) < n, so, by the inductive hypothesis, Pq = 0. This means that Pq is 
constant, and thus P = e"*Po is either a constant (if ct\ = 0) or an exponential 
function. In either case, P could not have had > n + 1 zeros. I 



An Example 

The simplest nontrivial example is the double-integrator system x = u, written 
in state-space form: m = 1, 11= [—1,1], n = 2, and matrices 

JJ J). »-(?). .-I, 
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This models a mass moving in one dimension, with no friction, and subject to a 
force of magnitude at most one in either direction. We know from the previous 
results that optimal controls are unique (up to sets of measure zero), and are 
bang-bang with at most one sign change (since n — 1 = 1). That is, any time- 
optimal control joining a pair of states must necessarily be either constant and 
equal to 1 or — 1 , or it must start at one of these values and then switch (once) 
to the other value. For this system, we will solve the problem of time-optimal 
control from any state x° to x ! = (that is, we will find the "optimal synthesis" 
that specifies optimal controls from each initial point to the origin). 
We have, explicitly, 

e -tA = (1 ~t 

\0 1 
so time-optimal controls all have the form 

uj(i) = sign(a-ci) (10.22) 

for some real numbers a and c which are not both zero (and lo(Tq) is arbitrary 
when a — cTq = 0, which can happen at most at one point To € [0, T]). So there 
are four possibilities for a time-optimal control Zj defined on an interval [0,T]: 

• u = — 1 



• for some To € i^jT), lu = —1 on [0,To) and u>(t) = 1 on (To,T] 

• for some Tq € (0,T), we 1 on [0,T)) and u>(t) = —1 on (To,T]. 

Conversely, any control of one of these four types, which steers a state x° into 
0, must be the unique time-optimal control doing so. This is because, first of 
all, any such control satisfies the maximum principle (just pick 7 = (c, a)' with 
a and c chosen so that a — ct has the right sign, switching at To in the two 
last cases), and second, because in this case (STLC, control to zero final state), 
the maximum principle is a sufficient as well as necessary test (cf. Proposi- 
tion 10.2.9). That is, any bang-bang control with at most one switch is optimal, 
for this problem. 

We let S be the set consisting of all those states which can be steered to 
x l = using constant controls. Thus S = S- (J S+, the sets of all points that 
can be controlled to zero using u = — 1 and u = 1 respectively. To find these 
sets, we solve the differential equations x = Ax — b and x = Ax + b respectively, 
both with final value x(t) = 0, obtaining two functions (— 00, 0] — * R 2 , £_ and 
£ + respectively: 

€-(*) = (-f), e + (*) = (f 

Hence S- is the half-parabola {(— #2/2, £2), #2 > 0} (in the second quadrant) 
and S + is the half-parabola {(a; 2 .^,^),^ < 0} (fourth quadrant). The set 

S = 5_|j5 + = {2 Xl + x 2 \x 2 \ = 0} 
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splits M 2 into two parts, one "under" S and the other one "over" S. 




Figure 10.1: Switching surfaces. 

We claim the following facts: 

• If the initial state a; is in S+, then the unique time optimal control to 
reach is obtained by applying u = 1. 

• If the initial state a; is in S-, then the unique time optimal control to 



reach is obtained by applying u 



-1. 



• If the initial state x Q is "over" the set S, then the unique time optimal 
control to reach is obtained by applying u = — 1 until the set 5+ is 
reached, and then switching to u = 1 . 

• If the initial state x° is "under" the set S, then the unique time optimal 
control to reach is obtained by applying u = 1 until the set 5_ is reached, 
and then switching to u = — 1. 

Recall that, for this example, every bang-bang control with at most one switch 
is optimal, so all we need to do is to show that if we start "over" the set 
S and apply u = — 1, the trajectory #(•) that results indeed crosses S+ (and 
analogously if we start under S). To establish this, one could compute solutions, 
but we can also argue as follows: since xi = — 1 < 0, eventually xi(t) becomes 
negative. (It could not happen that x(t ) € 5L for some t$, since in that case, 
solving the equation x = Ax — b backward from to gives that x(t) G S— for all 
t < to, because of the definition of S-, contradicting the fact that x(0) = x° is 
not in S.) Once that X2(t) becomes negative, we have that x\{t) = X2(i) < 0, 
so trajectories move in a "southwest" direction, with ±2 = —1, and hence must 
eventually meet S + . The case where we start under S is proved in the same 
manner. 
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Exercise 10.3.8 For the above example, take in particular the initial state 
x° = (—1,0)'. Show that the above synthesis results in the rule "accelerate 
at maximum power for the first half of the time and decelerate maximally for 
the second half". Derive this result, for that particular initial state, using 
elementary arguments not involving the maximum principle. □ 

10.4 Remarks on the Maximum Principle 

For fixed a; , we may define the Bellman or value function for the problem of 
time-optimal control from x . 

V{x l ) := min{T > | x f £ TZ T {x )} . 

This is a function TZ(x°) — > R. (Reversing time, we could also define an anal- 
ogous V for the problem of control to x.) Being defined by a minimum, this 
function tends not to be differentiable; but when it is, its gradient is a normal 
vector as needed for the maximum principle: 

Lemma 10.4.1 Assume that V is differentiable at some x f £ int 7\L(:e ). Let 
T := V(x { ). Then ry = \7V{x f )' is normal to TZ T {x°) at x { . 

Proof. We start by establishing the following claim: in general, if V : O — * 
K is a function defined on an open subset of K", C C W 1 is convex, and V 
achieves a local maximum relative to Cf]0 at some point £ £ Cf]0, then 
\7V(CY{z — £) < for all z £ C . Pick any z £ C ', and substitute the vector 
h = e(z— Q, for any fixed small enough e £ (0, 1], into the expansion V(C+ h) = 
V r (C) + VV r (C)'/i + o(||/i||). Since ( + h = ez+ {I - s)( £ C, if e is small enough, 
V{( + h)< V{() gives 

vy(0'(*-<) + 0(£||z " cll) < o. 

£ 

Letting e — > gives the claim. 

Now we apply this general fact with C = lZ T (x°) and O = vt\t1Z(x )] note 
that V achieves a local maximum at x l because V(z) < T = V(x°) if z £ 

n T (x°). ■ 

Remark 10.4.2 When V is not differentiable, it is possible to generalize the 
interpretation of VV^ by means of "viscosity" or "subgradient" terms. □ 

Exercise 10.4.3 (a) Consider again the double integrator example in Equation 
(10.21), with IX = [—1, 1], but now provide a solution to the problem of control- 
lability from x° = to an arbitrary state x l £ R n . Justify carefully why your 
solutions are indeed optimal, and make an illustration analogous to Figure 10.1. 
(b) For the problem in (a), let V(x l ) = min{T | x l £ 1Z T (0)}. Provide an 
explicit formula for the function V. What can you say about differentiability of 
VI □ 
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Remark 10.4.4 The maximum principle is not merely useful as a technique 
for finding optimal controls or for characterizing their properties. It is also 
an extremely useful tool for solving the apparently different problem of simply 
finding a control steering one state to another, because it allows one to restrict 
the search to a smaller and often more manageable class of controls than all of 
C°° . Suppose that we wish to find a control lu, if one exists, that steers a given 
initial state a; to a desired target state x . Assuming appropriate technical 
hypotheses of normality as discussed earlier, if there is a control that works, 
then there will be a time-optimal one lu, which satisfies the maximum principle. 
Define the function m : K™ — > U by the formula m(p) := argmax ueli p / Bu. 
Then, we know that there must be some (as yet unknown) nonzero vector 7 G W l 
so that, solving the system of 2n differential equations 

x = Ax + Bm(X), x(0) = x° 
A = -A'X, A(0)=7, 

one obtains that x(j) = x { for some r. Now, a numerical search can be carried 
out over the space of possible initial adjoint vectors 7, solving this differential 
equation, and testing along solutions whether \\x(t) — x ! \\ becomes smaller than 
a certain error tolerance. Thus the search has been reduced from a search in 
an infinite-dimensional space to searching over a space of dimension n, or, even 
better, n — 1 (since, without loss of generality, one could take 7 to belong to the 
unit sphere in R"). There are technical difficulties to be dealt with, including 
the fact that the right-hand side of the equation may not depend continuously 
on A (because m may not), but the idea is nonetheless very powerful. A related 
application of the same idea is as a general technique for estimating the bound- 
aries of reachable sets TZ T (x°), which can proceed in the same manner (since 
x(t) G TZ l {x ) when solutions have been generated in this manner). 

A different technique, motivated by bang-bang theorems, is to look for bang- 
bang controls, parametrized finitely by switching times, assuming an a priori 
bound is known on the number of switches required (as, for instance, in the 
cases covered by Proposition 10.3.7). □ 

10.5 Additional Exercises 

In all exercises, we assume given a linear time-invariant continuous time system 
x = Ax + Bu and IX compact convex. 

Exercise 10.5.1 Assuming STLC holds, let the minimal time function V be 
defined as above, for reachability from the initial state a; = 0. Prove: the level 
sets {V(a;) = T} are precisely the boundaries of the reachable sets 1Z T (0). □ 

Exercise 10.5.2 Let C T {x f ) := {x° \ x f G TZ T {x )} (set of states that may be 
steered to x i in time T and let C(x ) = \^j T>0 C T (x i ). Prove: 

• If x° G C T (x ) and T is minimal with this property, then a; G dC T (x ). 
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• Assume that x° G dC T (0) and that the STLC is satisfied. Then T is 
minimal with the property that x° G C T (0). □ 

Exercise 10.5.3 Let lZ- T (x°) := [j 0<t<T TZ t (x ). Show that, for any x and 
T, 7^ T (x°)is: 

• connected, 

• compact, 

• but not necessarily convex. 

(Hint: (For compactness.) If x k G 1Z tk (x°), you may assume that tk\T or 
t k \T for some T (why?). Then, restrict or extend controls u>k to [0,T]. Finally, 
use a compactness argument as in the proof that 1Z (x°) is compact.) □ 

Exercise 10.5.4 For any metric space M, we use K(M) to denote the family 
of all nonempty compact subsets of M, and define 



D(Ci,C2) '■= max < max d(x, C2), max d(x, C\) 
(left xgc 2 

Show that D defines a metric on K (usually called the Hausdorff metric) . Now 
consider a linear system x = Ax + Bu and a fixed initial state x Q G R™ . Show 
that the mapping T 1— > 1Z (x°) is continuous as a map from R into K(R"). □ 

Exercise 10.5.5 Consider the undamped harmonic oscillator x\ = X2, X2 = 
—x\ + u, with control-value set U = [— 1, 1]. Show, for each two states x° , x l : 

1. there is a unique control ui steering x° to x l in minimal time, 

2. this control has |u(i)| = 1 for almost all t, and 

3. the intervals between sign changes have length 7r. 

Finally, provide a complete description of the optimal controls for the prob- 
lem of transferring any state to x f = 0, including a diagram. (Hint: For the 
controllability part, recall Exercise 3.6.8.) □ 

Exercise 10.5.6 Consider the system x\ = X\ + u, x 2 = x 2 + u, with control- 
value set 11 = [—1,1]. Take x° = (1, 0)' and x f = (2, 0)'. What is the minimum T 
so that x l G TZ T (x°)7 Show that the maximum principle provides no information 
whatsoever about a time-optimal control steering x° to x l . (What property 
fails?) Show that there are, in fact, infinitely many controls steering x° to x l in 
minimal time. □ 

Exercise 10.5.7 Consider this nonlinear system, with X = R 2 and U = [—1,1]: 

X\ = (1 — x\) u 2 

±2 = U . 

Show that the system is complete, that is, solutions exist for every initial state 
and every control. Show, however, that ^(O) is not a closed set. Why does this 
not contradict the discussion in Remark 10.1.10? (Hint: Consider the states 
x{ = a(u>k), where uj k is a control that switches fast between +1 and — 1.) □ 
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Exercise 10.5.8 (a) Consider a controllable single-input system x = Ax + bu. 
Show that there is some 5 > with the following property: for each 7^0, the 
function t i— > 7'e - * b has at most n — 1 zeros in the interval [0, 5}. 

(b) Conclude that, for controllable single-input systems, with IX = [u, u], there 
is some 6 > so that, whenever x° and x l are so that x l can be reached in 
time at most 5 from x°, the time optimal control steering x° to x l is piecewise 
constant, taking values u or u in at most n intervals. 

(c) Conclude further that, if T is the minimum time for steering x° to x { , then 
the time optimal control steering a state x° to a state x is piecewise constant, 
with at most Tn/S switches. 

(Hint: (For (a).) Assume the property is false on [0,1 /A;]. Note that one may 
restrict attention to 7's so that ||7|| = 1. What can be said about the zeros of 
derivatives of "/e~ tA bl Take limits k — > 00.) □ 

10.6 Notes and Comments 

Weak convergence as defined here is a particular case of a concept that is stan- 
dard in functional analysis. In general, if N is a normed space and N* is its 
dual (the set of continuous linear functionals N — » R), the weak-star topology 
on N* is the topology defined by the property that a net {x^.} in N* converges 
to x* G N* if and only if x* k (x) — > x* (x) for all x G N. In particular, we may 
take N = £^(0,T), the set of integrable functions ip : [0,T] -> R m with the 
norm L ||w(s)|| (is. The space £^?(0,T), with the norm IHI^, can be naturally 
identified with the dual of N via 

f T 
u>(ip) := / y(s)'w(s) (is . 

./o 

Thus, a net cj^ (in particular, a sequence) converges to u) in the weak-star sense 
if and only if L ip(s)'u>k{s) ds — > /„ </p(s)'w(s) (is for all integrable functions 
ip : [0,T] — > R m . So what we called weak convergence is the same as weak- 
star convergence. Alaoglu's Theorem (see e.g. [98], Theorem V.3.1). asserts 
that the closed unit ball in N* (or equivalently, any closed ball) is compact in 
the weak-star topology. This proves Lemma 10.1.2. However, we provided a 
self-contained proof. 

In connection with Remark 10.1.12, we note that it is possible to relax the 
convexity assumption; see [188], which may also be consulted for many more 
results on the material covered in this chapter. 

In connection with Remark 10.4.2, the reader is directed to [93]. 
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Linear Algebra 



A.l Operator Norms 

If x € C , we introduce the Euclidean norm 



If A is a complex n x m matrix, by \\A\\ we denote the operator norm 

\\A\\ := max \\A^\\ (A.l) 

IMI=! 

with respect to the above norm on C™ and C m . 

Numerically, \\A\\ is just the largest singular value of A, that is, the square 
root of the largest eigenvalue Ai of the Hermitian matrix 

Q := A* A 

(where the star indicates conjugate transpose), as discussed later in this Ap- 
pendix. If v\ is an eigenvector corresponding to Ai, then ||A«i|| = ||j4|| ||vi||. In 
particular, when A is a real matrix, vi can be chosen real, so the same value 
1 1 A 1 1 is obtained when maximizing just over real vectors fi such that ||/x|| = 1. 

It is a basic fact from linear algebra that there are positive constants c±, c%, 
which depend only on n, m, such that 



ci /£4 < \\a\\ < C2 JE4 ( A -2) 

for all such A. 

If {A{\ is a sequence of matrices (all of the same size n x m), we define 
convergence 

Ai^B 
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to mean that \\Ai — B\\ — > 0. Equivalently, because of (A. 2), this means that 

\Al)ij '' £>ij 

for each i,j. 

A. 2 Singular Values 

The concepts of singular value and singular value decompositions are very useful 
in linear control theory. We develop here the basic facts about these. 

Let A be a complex n x I matrix of rank r. Using "*" to denote conjugate 
transpose, let Q := A* A. Since Q is Hermitian and positive semidefinite, it 
admits an orthonormal basis of eigenvectors, 

Qvi = XiVi, i = l,...,l, (A. 3) 

where the eigenvalues Ai are real, 

0< A, < A,_! < ... < Ai. 

Note that rank Q = r, so A r +i = . . . = A; = 0. 

We define c, := y/Xi for each i = l,...,r. By definition, these are the 
singular values of the matrix A. In particular, one often denotes ay as 

O'min(A) , 

and this is the smallest singular value of A. When A has full column rank — that 
is, ker A = 0, r = I, and Q is nonsingular — \\Ax\\ = x*Qx > cr^ \\x\\ , so 

||Ac|| > a min (A) ||x|| (A.4) 

for all vectors x; so cr m i n (yl) provides a measure of how far A is from being 
singular, in the sense of having a nonzero kernel; in fact, as proved below, 
this number measures the precise distance to the set of singular matrices. As 
1 1 Ac 1 1 = x*Qx < erf \\x\\ for all x and ||j4vi|| = erf \\vi\\ , the largest singular 
value <J\ is the same as the operator norm of A with respect to Euclidean norms 
on vectors, as discussed above in Section A.l. 
Then, we introduce 

m := — Avi G C" , i = 1, . . . , r , 



and note that 



1 *n X J 

■ — v i Qvj = — - 



for each i,j between 1 and r, and therefore that the set 

{ui,...,u r } 
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is orthonormal. By the Gram-Schmidt procedure, we complete this to an or- 
thogonal basis of C n and let 

U := (wi,. . .,u„) 

be the unitary matrix that results from listing all elements of that basis. Note 
that, for j < r and any i, 

[ if i =£ j 

U 4 AVj = (7-jUj Uj = < 

7 i 



' ' " ' ' 1 Uj otherwise, 



and that for every j = r + 1, . . . , I the product is zero since Vj is in the kernel of 
A. Thus, if we define V to be the unitary matrix whose columns are the vectors 
Vj, j = 1, . . . , I, then U* AV = E, where £ is the matrix 

r I — r 

\ 

(A.5) 








o J 

that is, an n x / matrix whose only nonzero entries are the first r principal 
diagonal entries Y,a, and these are precisely the singular values of A. One calls 
the equality 

A = UT.V* (SVD) 

a singular value decomposition (SVD) of A. From this, one can also write 
A as a sum of rank one matrices 

r 

a = y^ aiUiv* , 



and this sum is itself sometimes called the SVD of A. Note that for real matrices 
A, both U and V can be taken to be real, that is, orthogonal matrices, since the 
constructions can all be done in that case over JR. Singular value decompositions 
can be computed efficiently and reliably using any of the standard numerical 
packages such as LINPACK. 

Exercise A. 2.1 Assume that there exist unitary matrices U, V so that A = 
U"EV* , where £ is as in (A.5). Prove that the aVs must be the singular values 

of A. a 

Lemma/Exercise A. 2. 2 Prove that for every matrix A: (i) A, its conjugate 
transpose A* , and its transpose A' all have the same singular values; and (ii) 
for every nonzero A s C and every i, <7j(A^4) = |A| <Ji{A). □ 
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One simple application of SVDs is in computing the Moore-Penrose inverse 
or pseudoinverse of A. (Also called the generalized inverse.) This is the matrix 



where £# is the I x n matrix 



A* := VY^U* 



(A.6) 



I 



\ 







o J 



whose only nonzero elements are the inverses of the singular values of A, placed 
on the main diagonal. The pseudoinverse solves the general least-squares prob- 
lem of finding the vector x of minimum norm among all those vectors minimizing 

\\Ax-b\\ . 

We claim that x := A#b is the (unique) such minimal solution. Indeed, given 
any x and b, 

\\Ax-bf = \\Xy-cf, 

where y := V*x and c := U*b; since x and y have the same norm, because V 
is unitary, it is equivalent to minimize as a function of y and then to substitute 
again x. But this expression is just 



4=1 



WiVi 



Ci\ 



i=r+l 



which is minimized when yi := Ci/(7i for i = 1, . . . , r. The other coordinates of 
y have no influence. Among all such vectors, minimum norm is achieved when 
setting all other coordinates of y to zero, that is, 



y 



S*U*b, 

from which it follows that x = Vy = A#b, as claimed. 

In the text we deal with pseudoinverses of operators between an infinite 
dimensional space and a finite dimensional one, under a full rank assumption 
that results in a simpler formula for the pseudoinverse; the next exercise provides 
the connection. 

Exercise A. 2. 3 Assume that A has full row rank r = n. Show that, then, the 
formula for the pseudoinverse simplifies to 

A* = A*(AA*)~ 1 , 

which can be evaluated without computing singular values. (What is the analo- 
gous formula when A has full column rank?) In particular, for square matrices 
of full rank, A* = A' 1 . □ 
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As remarked above, cr m i n (A) measures how far the matrix A is from being 
singular. We now make this precise. As usual, ||-|| denotes operator norm. 

Lemma A. 2. 4 For any matrix A of full column rank r = I, 

a min (A) = dist(A,sing) :=min{||A|| | ker {A + A) ^ 0} . 

Proof. Note that the minimum in the right-hand side of the claim is achieved, 
since matrices B with nonzero kernel form a closed set (for instance, defined 
by the vanishing of the determinant of B*B). Let A and x ^ be such that 
||A|| = dist(A,sing) and (A + A)x = 0. We write A = A + A - A and apply 
(A. 4) to get: 

*min(A)\\x\\ < \\(A+A)x\\ + \\A\\\\x\\ = \\A\\ llxll, 

from which it follows that 

Vmin{A) < || A|| =dist(A,sing). 

Conversely, consider the matrix 

A := -<7 min (A)u r v* 

obtained by replacing all other singular values in (SVD) by zero. Since 

r-l 

A + A = ^ (JiUiV* 

i=\ 

has rank less than I, 

dist(A,sing) < || A|| =<Jmin(A) 

and the equality follows. I 

This result is used in the text in order to obtain characterizations of how 
uncontrollable a system is. From Lemmas A. 2. 4 and A. 2. 2 we also know that: 

Corollary A. 2. 5 For any matrix A of full row rank r = n, 

a min (A) = min{|| A|| | (A + A) is not of full row rank } . □ 

A result in a latter section of this Appendix establishes that the singular 
values of a matrix, and in particular c m j n (A), are continuous functions of the 
entries of A, for matrices of any fixed rank. 
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A. 3 Jordan Forms and Matrix Functions 

Jordan forms are not very useful numerically but are convenient when estab- 
lishing various results. We work here with matrices over C, but the existence of 
Jordan forms can be proved over arbitrary algebraically closed fields. 

Recall the basic result, which can be found in any elementary linear algebra 
book. (We state this in a weaker form than usual; the complete result includes 
a uniqueness statement, but this is not needed in the text.) 

Proposition A. 3.1 For any matrix A G C" x ™ there exists an invertible n x n 
complex matrix T and there is some integer I > 1 such that T~ 1 AT has the 
block structure 









Vo 





■h 









Jl-1 











jj 



(A.7) 



where each block J, is a fc x ki matrix of the form 



(\ 1 
A 1 






Vo 



... o 
... o 





... 1 





... \ 

... o 


1 



(A. 



for some eigenvalue A = A^ of A. □ 

A matrix of the form (A. 8) is a Jordan block (corresponding to the eigenvalue 

A). 

Note that the integers fcj must add up to n. The same eigenvalue of A may 
appear in more than one block, and the multiplicity of each eigenvalue A is the 
same as the sum of those integers ki for which Aj = A. Note that the only 
possible eigenvectors of each Jordan block J are those of the form 



(o,0,...,0) / 



(A.9) 



(prime indicates transpose) , so for each eigenvalue A of J (equivalently, of the 
original matrix A) the geometric multiplicity of A, that is, the dimension of the 
A-eigenspace 

kev(XI-A), (A. 10) 

is equal to the number of blocks corresponding to A. 

Let / : C — > C be any entire function, that is, a function analytic on the 
entire plane, and let 



/(*) = £■ 



i=0 
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be its expansion at the origin (which has an infinite radius of convergence). For 
any square matrix A, we define f(A) by the series 

oo 

f{A):=Y,aiA\ 



This series is convergent, because the inequalities 

ll^ll < \\A\t 
imply that the sequence of partial sums is a Cauchy sequence in C nx ": 



X>A' 



t—p 



i—p 



and the right-hand side goes to zero because 

DC 

^\ai\r l < oo 



i=0 

for all real numbers r. 

Assume that / and g are entire and consider the entire function h := / • g. 
Since the Taylor series for h about z = can be computed as the product of 
the series of / and g, it follows by substitution that 

(fg)(A) = f(A)g(A) (A.ll) 

for all matrices A. Since fg = gf, it follows also that 

f(A)g(A) = g(A)f(A) 

for all A and all functions. (Using an integral formula, it is possible to define 
f(A) for functions / that are not entire but merely analytic in a neighborhood 
of the spectrum of A, in such a way that these properties hold, but we do not 
need that more general definition here.) 
The following is very easy to establish. 

Lemma/Exercise A. 3. 2 Let / be an entire function and assume that J is of 
size k x k and has the form (A. 8). Then, 



f(J) 



/"(A) 
2! 

/(A) /'(A) 



//(A) /'(A) T 






V o 



f {k - 2, W 

(fe-2)l 

/ ( "- 3) (A) 

(fe-3)l 



/'(A) 

/(A) 




/ (fc - 1; (A) 
(fc-l)l 

/ ( "- 2) (A) 
(fe-2)! 



/"(A) 
2! 

/'(A) . 



(A.12) 
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As a consequence, the Spectral Mapping Theorem is proved: If A is any matrix 
with eigenvalues 

Ai, . . . , A n 

(counted with repetitions), then the eigenvalues of f(A) are the numbers 

/(Ai),...,/(A„) 
(also with repetitions). □ 

If v is an eigenvector of A, Av = Xv, then A l v = X l v for all i, so f(A)v = 
f(X)v and v is an eigenvector of f(A), too. From this, or from the Jordan 
form, we conclude that the geometric multiplicity of /(A) with respect to f(A) 
is greater than or equal to that of A with respect to A. 

Note that, if B = T^AT, then also B 1 = T- 1 A t T for all i, so 

B = T^AT => f{B) = T~ 1 f{A)T (A. 13) 

for all A, all entire /, and all invertible T. Together with Proposition A. 3.1 
and Lemma A. 3. 2 this gives a way of computing f(A) if one has obtained the 
Jordan form (A. 7): f(A) is of the form TBT" 1 , where B is a block matrix each 
of whose blocks has the form (A. 12). In particular, it follows that f(A) = g(A) 
if it is only known that / and g, and their respective derivatives of high enough 
order, coincide on the spectrum of A. 

A particular case of interest is f(z) = e tz for any fixed t. Let A be any fixed 
(square) matrix. As a function of t, 



e tA = V — -t m A m (A. 14) 



m— 



converges uniformly on compacts, since 

oo 1 



y —t m A m 

*—* m! 

m=k 



< 

*-^ ml 



Similarly, taking termwise derivatives of (A. 14) results in the series for Ae , 
which is also uniformly convergent on t. It follows that 

de tA , , 4 

— = Ae tA , (A.15) 

so e solves the matrix differential equation X = AX. From Lemma A. 3. 2 
together with (A. 13) it follows that each entry of e is a linear combination of 
terms of the form 

t s e xt , (A.16) 

where s is a nonnegative integer and A is an eigenvalue of A. 

The following fact is of use when discussing controllability and observability 
under sampling. 
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Lemma A. 3. 3 Assume that / is an entire function that is one-to-one on the 
spectrum of A, and let A be an eigenvalue of A for which /'(A) ^ 0. Then the 
geometric multiplicity of /(A) with respect to f(A) is the same as that of A 
with respect to A. Moreover, the /(A)-eigenspace of f(A) is the same as the 
A-eigenspace of A. 

Proof. We may assume that A has the form (A. 7). Since eigenvectors corre- 
sponding to /(A) are sums of eigenvectors corresponding to the various blocks 
/(Jj) for which Ai = A (by the Spectral Mapping Theorem plus the one-to-one 
hypothesis), it is enough to show that the geometric multiplicity of /(A) is one 
for the block (A. 12), under the hypothesis that /'(A) ^ 0. To show this, it is 
enough to see that there cannot be any eigenvector that is not of the form (A. 9). 
But the equation 



f(J) 



a 2 



\a k J 



/(A) 



a 2 



\a k J 



implies (by induction, starting with the (k — l)st row) that 



OLk = CXk-1 



«2 = . 



as desired. 



Actually, a stronger result holds. If / is as in the Lemma and /' does not 
vanish at any eigenvalue of A, then A is a function g(f(A)) for some analytic 
function g. This can be proved using the Implicit Function Theorem, but re- 
quires the definition of analytic (not necessarily entire) functions of a matrix. 

Recall that a cyclic n x n matrix F is one for which there exists some vector 
v £ C™ (a cyclic vector for F) so that 



span {v, Fv, . . . , F v, . . . , } 



(A.17) 



If F is cyclic, then translating it by a multiple of the identity does not affect 
cyclicity, that is, for any real number a the matrix F + al is again cyclic. This 
is because, for each k and each vector v, 

span {v, Fv, . . . , F v} = span {v, Fv + av, . . . , F v + kaF ~ v + . . . a v} . 

(A.18) 
Observe that any Jordan block J is cyclic, using the vector (0,0,..., 0, 1)'. 



Exercise A. 3. 4 Let / be an entire function and assume that J is of size k x k 
and has the form (A. 8). Assume further that /'(A) ^ for each eigenvalue A of 
A. Prove that then /(J) is cyclic. □ 
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A. 4 Continuity of Eigenvalues 

The fact that eigenvalues depend continuously on matrix entries is essential in 
understanding many linear systems results, especially those dealing with ro- 
bustness, and is useful as well as a technique of proof for matrix identities. In 
this section we provide a precise statement of this fact. Since eigenvalues of A 
are zeros of the characteristic polynomial of A, and in turn the coefficients of 
this characteristic polynomial are continuous functions of the entries of A, it is 
sufficient to prove the continuous dependence of zeros on the coefficients of a 
(monic) polynomial. 

For each complex number A and real e, we let B e (X) (respectively, S e (X)) be 
the open disk (respectively, circle) of radius e centered at A. 

Lemma A. 4.1 Let f(z) = z n + a\z n ~ x + . . . + a n be a polynomial of degree n 
and complex coefficients having distinct roots 

Ai, . . . , \g , 

with multiplicities 

m + . . . + n q = n , 

respectively. Given any small enough e > there exists a 5 > so that, if 

g(z) = z n + b 1 z n ~ 1 + . . . + b n , \a,i-bi\<6iori = l,...,n, (A. 19) 

then g has precisely nii roots in B e (Xi) for each i = 1, . . . , q. 

Proof. We assume without loss of generality that e is smaller than all dif- 
ferences |Ai — Aj| /2 for all i ^ j, and denote Si := S e (Xi) for each i. These 
circles are disjoint and the interiors contain exactly rii roots of / counted with 
multiplicities, so from the Residue Theorem 

1 f ?&& 



27ri J s t f( z 

(the integral evaluated counterclockwise) for each i. 

Consider the polynomial g in (A. 19) as a function g(b, z) of b = (bi, . . . , b n ) 
and z. As such it is continuous, and the same holds for 3 i 'V provided that 
g(b, z) ^ 0. By uniform continuity when z is on the closure of the union of the 
disks B e (Xi) and 6 is near a = (ai, . . . , a„), it follows that g(z) ^0 on every Si 
and 



/'(*) 9'{z) 



1 
< - 

e 



f(z) g(z) 

when g is as in (A. 19) and 5 is sufficiently small. But then 



1 f 9(z) , 
— / V-f-fdz - m < 1, 



2m J s g(z) 
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which implies that this integral — which is an integer that counts the number 
of zeros of g in B e (Xi) — is also equal to rii, and the conclusion of the Lemma 

follows. I 

We now list a few useful but very simple corollaries. The first is just a 
restatement of the Lemma. 

Corollary A. 4. 2 Let U be any open subset of C and fix any two nonnegative 
integers k < n. Then the set of monic polynomials of degree n having at least 
k roots in U is open, when polynomials are topologized by their coefficients as 
above. □ 

Corollary A. 4. 3 Let n be any fixed positive integer. For each a = (ai, . . . , a n ), 
let 

n(a) < r 2 {a) < ... < r n (a) 

be the real parts of the zeros of P a {z) = z n + aiz" -1 + . . . + a n arranged in 
nondecreasing order. Then, as a function C" — » R, each r, is continuous. 

Proof. Fix any a and let rj := ri(a) for each i. Let Ai, . . . , \ q be the distinct 
zeros of P a , with multiplicities m, . . . , n q , and arranged so that their real parts 
are nondecreasing. Pick any 

f |ReAi-ReA,-| „ v , „ , 

£ < min <^ J — ^ Re A 4 ^ Re Xj 

and find 5 as in Lemma A.4.1. We claim that 

\ri(a) - n(b)\ < e 

for each i whenever all |a» — 6,| < 6, which will prove continuity. 
Fix any i = 1, . . . , q, and observe that 



'ni-i + l 



(a) = ... = r ni (a) = Re A 4 ( A.20) 



(denoting no := 0). For b close to a, the polynomial P^ has exactly rii roots at 
distance less than e from A^ , so the real parts of these m zeros are at distance 
less than e from Re Aj. 

Assume that Re A^ > Re Aj for j = 1, . . . , a, a < i, and that Re A^ < Re Aj 
for j = /?,..., q, for some [3 > i. From the choice of e and the fact that these 
real parts are at a distance less than e from ri(a), . . . , r ria (a), it follows that 

n(b),...,r noi (b) 

are at distance larger than e from r n .(a), and the same is true for r,j(b), j > (3. 
Thus, the zeros within distance e of Re A^ are precisely r„ Q+1 (6), . . . , r„„_i(6), 
and in particular 

\rj(b)-rj(a)\ <e 
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for j = rii-i + 1, . . . ,n». This shows that Tj is continuous, for each such j. Since 
i was arbitrary, the result holds for all r^'s. I 

The same proof of course is applicable if instead of real parts one looks at 
magnitudes of zeros. 

Corollary A. 4. 4 For each positive integer n, there exist n continuous functions 

A 1 ,...,A n :C nxn -R 

such that Xi(A) < . . . < X n (A) for all A, and if A is Hermitian, Ai(j4), . . . , X n (A) 
are the eigenvalues of A. 

Proof. Just apply the previous corollary to the characteristic polynomial and 
use that the eigenvalues are all real when the matrix is Hermitian. I 

Corollary A. 4. 5 For each pair of positive integers I and n, there exist I con- 
tinuous functions 

ai,...,<rj :C" X '^K 

such that for each A of rank r < I, cr r (A) < . . . < &i{A) are the singular values 
of A. In particular, on matrices of any fixed rank r, <r m i n (A) is a continuous 
function of A. 

Proof. Just let (Ti(A) := \/A;_j+i(A*^4), where the Xi are the functions in 
Corollary A. 4. 4. I 

As a different example of the use of Lemma A. 4.1, consider the following 
fact, which is used in the text. It is typical of the use of continuous dependence 
in order to reduce questions about arbitrary matrices to simpler cases. 

Proposition A. 4. 6 Let A and B be two nx n complex matrices, and consider 
the linear operator 

La.b ■ <C nxn -> C nx " : X h-> AX - XB' . 

Then each eigenvalue of La,b is of the form 

X — fi 

for some eigenvalues A and /x of A and B, respectively. 

Proof. First consider the case of matrices A and B for which all of the possible 
n 2 differences 

A, - hj (A.21) 

are distinct, where Ai, . . . , A„ and /xi, . . . , )J, n are, respectively, the eigenvalues 
of A and B. In particular, each of A and B has distinct eigenvalues. In this 
case there exist two bases of eigenvectors 

Av l = X i v i , Bw l = pL l w i , i = l,...,n. 



A. 4. Continuity of Eigenvalues 459 

Let Xij := Viw'j for each pair i,j. We claim that these are eigenvectors of La,b 
with corresponding eigenvalues A^ — \Xj. Indeed, 

AX tj - X i: jB' = Av^ - Vi(Bwj)' = AjUji^- - v^HjUJj) 1 = (A; - Hj)X i:j . 

Since there are n 2 such eigenvalue/eigenvector pairs, every eigenvalue has the 
required form. 

We now use continuity on eigenvalues in order to obtain the general result. 
Given any two matrices A, B, assume that there is some eigenvalue a of La,b 
that is not of the form X — fi. Let e > be such that 

\a - (\ - n)\ > e (A.22) 

for all eigenvalues A and /i of A and B, respectively. Pick 5 > so that, if 
\\A — A\\ < 5 and \\B — B\\ < 6, then all eigenvalues of A (respectively, B, Lr ~) 
are at distance less than e/3 from eigenvalues of A (respectively, B, La,b)- 
The Lemma is used here; note that any fixed matrix representation of La,b is 
continuous on A, B. 

It is always possible to find a pair (A, B) so that \\A— A\\ < 5 and \\B— B\\ < 5 
and all of the n 2 differences A^ — Jij are distinct. (Just perturb the Jordan forms 
for A and B.) Using one such pair and applying the conclusion in the special case 
proved first, there must exist some eigenvalues A and Ji of A and B, respectively, 
so that 

a — (A — Jj) < e/3 , 



while at the same time 



A- A 



< e/3 and \(x — jj\ < e/3 for some eigenvalues of 



A and B, respectively. This contradicts (A.22). I 

Corollary A. 4. 7 The operator La,b is nonsingular if and only if A and B have 
no common eigenvalues. 

Proof. By the Proposition, nonsingularity is equivalent to asking that A — /i ^ 
for all eigenvalues A of A and /i of B. I 

Remark A. 4. 8 There is another proof of Corollary A. 4. 7 that does not re- 
quire explicitly the result on continuity of eigenvalues. This is based on the 
observation that the determinant of La.b is the same as the resultant of the 
characteristic polynomials of A and B, and hence the determinant is zero if and 
only if these two polynomials have a common root, that is, if A and B share a 
common eigenvalue. 

The fact about the resultant is in turn proved as follows. For matrices 
A, B so that all the numbers A^ — fXj are distinct, the first part of the proof 
of Proposition A. 4. 6 shows that the determinant is the product of all of the 
terms (A^ — fij), that is, the resultant. Since the resultant is a polynomial — and 
therefore continuous — function of the entries of A and B, and pairs of matrices 
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A, B so that all of the (Aj — /Uj)'s are distinct form a dense subset, the result 
follows for all matrices. 

This technique is different, and it has the advantage that it extends to arbi- 
trary fields, and in fact arbitrary commutative rings. That is, over any commu- 
tative ring R the determinant of L^ g, seen as an operator on n x n matrices 
over R, is the resultant of \ A an d X R - This is because any polynomial identity 
on matrices that is proved for complex matrices is necessarily true over arbitrary 
commutative rings. (Using some algebra: It is only necessary to restrict to the 
subring generated by the entries of all matrices appearing in the identity; this 
is a quotient of an extension of finite transcendence degree of Z and is therefore 
a subring of C. So identities over C project into identities over the ring.) □ 

Exercise A. 4. 9 Prove the Cay ley-Hamilton Theorem for complex matrices by 
first reducing it to diagonal matrices and using continuity of characteristic poly- 
nomials and invariance of the Theorem's conclusions under matrix similarity. 
(As mentioned in Remark A. 4. 8, this also automatically establishes the Cayley- 
Hamilton Theorem over arbitrary commutative rings.) □ 



Appendix B 

Differentials 



This Appendix reviews basic notions of differentiability, first for multivariate 
maps and then for maps between normed spaces. The latter are used in the 
text in order to characterize linearizations of continuous-time systems. 



B.l Finite Dimensional Mappings 

We first recall some basic concepts from multivariable calculus. A mapping 

T=\ : I :U-> V (B.l) 



defined on an open subset IX C K m and with values in an open subset V C W l 
is said to be k-times differentiable or is of class C k if each of its components 

T % : IX -» R 

has continuous partial derivatives of order up to k. In the case k = oo, one says 
that T is smooth; when k = 1 one just says differentiable. When fc = 0, one 
means simply that the map is continuous. When the set IX is not necessarily 
open, we define smoothness to mean that there is a smooth extension to some 
neighborhood of IX. 

Given T as in (B.l) but over the complexes, that is, with IX C C m and 
V C C™, we may also see T as a 2n- vector of real- valued maps, depending on 
2m real variables. We shall say that such a T is of class C k if it is so as a map 
over the reals. This should not be confused with analyticity, which corresponds 
to differentiability as a function of m complex variables. Everything in this 
section will apply to systems over C by consideration of the real- valued systems 
of twice the dimension. 
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The Jacobian of the differentiable map T at the element u° G IX is the matrix 

dT t {u) 



T.[«°] 



duj 



which is continuous as a function of u . Continuity of the matrix means that 
each entry is continuous, or equivalently continuity of T as a map from 11 into 
K nxm , when the latter is given the topology that results from listing all coor- 
dinates and identifying SJ lxm with K" m . If n = 1, the Jacobian is the same as 
the gradient 

-M^> Sy») 

of T = T 1 . 

The chain rule states that the Jacobian of the composition is the composition 
(matrix product) of the Jacobians: 

(T o S)4u } = T4S{u°)}S4u } 

whenever T and S are differentiable and S(u ) is in the domain of T. 

One may also think of T* [u ], for each fixed u , as a linear mapping from R m 
to R™, the differential of T at u . Thus, the Jacobian is nothing more than 
the matrix of the differential with respect to standard bases. One advantage of 
considering the differential is that it can be defined naturally in infinite dimen- 
sional spaces, and we do that below. The generalization will follow from the 
following characterization of Jacobians, which can be found in any elementary 
analysis book (see, e.g., [121]): 

Fact B.l.l The map T is differentiable if and only if there exists a continuous 
function 

J : IX -> E" xm 

such that, for each u° G IX, 

lim ||r(,.)-r(„»,-i( : °)(-,,»,|| =o 

If such a J exists, then necessarily J{u ) = T*[u ]. □ 

We are using the notation ||cc|| for the Euclidean norm 

IMI = 

of a vector in Euclidean space. 

More generally, one applies these concepts for maps between infinite dimen- 
sional spaces as follows. 
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B.2 Maps Between Normed Spaces 

Recall that a normed space N over IK = R or C is a vector space over K, together 
with a map 

N -»R, in ||x|| , 

which satisfies the following properties for all x, z G iV and fc G K: 

1. ||x|| > Oif x^O and ||0|| =0 . 

2. ||MI = |*| INI • 

3. ||* + 2/II < Nl + Nl. 

A normed space N is a metric space when distance is defined as 

d (x, z) := ||x — z\\ . 

If N is complete under this metric, it is a Banach space. 

The notion of derivative, and more generally that of a Jacobian of a multi- 
variable map, can be generalized to maps between normed spaces. This gener- 
alization is based on the interpretation of a Jacobian of a map from IK m into IK" 
at a point x° as itself a map from IK m into IK n , and it satisfies the basic prop- 
erties of derivatives, such as the chain rule. We review here the basic definition 
and the main properties that we shall need. More details can be found in any 
modern analysis book, e.g., [121]. 

A continuous mapping 

F : O -» N 2 

from an open subset O of a normed space N\ to another normed space N 2 is 
(Frechet) differentiable at a point x° G O iff there is a linear mapping 

F* [x°] : TVi -» iV 2 

such that 

||F(x)-F(x°)-i=;[x ](x-x )|| = o(||x-x°||) 

as x — x° — > 0. If such a map exists, it is unique and is a bounded operator, 
called the differential of F at x . For each v G N\, F*[x ](v) is the directional 
derivative of -F at x° in the direction of u. 

If i 7, has a differential at each point of O, the derivative of -F is the mapping 

F, : O^L(N u N 2 ), x » F,[x] , 

where L(Ni, N2) is the set of all bounded linear maps from N± to N2 endowed 
with the operator norm. One says that F is of class C 1 if this mapping is 
continuous. Second derivatives of F are defined via derivatives of F*, and so on 
inductively. 
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Remark B.2.1 There is another notion of differentiability which is often use- 
ful. The mapping F : O — > -/V2 is Gateaux differentiable at a point x° G O if 
there is a linear bounded operator F*[x ] : N± — > A^ such that, for each v € Ni, 

\\F(x° + hv) - F(x°) - F,[x°](hv)\\ = o(h) (B.3) 

as h — » 0. If F is Frechet differentiable, then it is also Gateaux differentiable, 
with the same -F*^ ]. But Gateaux differentiability is weaker, because the limit 
is taken along each direction x — x° = hv, h — > 0, separately. Equation (B.3) 
says that the mapping 

P : X -> N 2 : h i-> F(a; + to) 

(defined on some interval (—ho, ho) about zero) is differentiable at h = 0, and 
has /3(0) = -F*[:c ](</). □ 

Provided that dim N\ > dim _/V 2 , we say that the map _F /ias /mW rank at x° 
if i 7, is a submersion there, i.e., if it is differentiable and -F*[a; ] is onto. 

We shall need to apply the Implicit Function Theorem to differentiable map- 
pings. In general, various technical conditions are needed in order for the Theo- 
rem to hold for maps among infinite dimensional spaces. However, in the cases 
of interest to us, one of the spaces is finite dimensional, so there is no difficulty. 
We start with the Rank Theorem. 

Theorem 52 Assume that F : O — » ./V2 is of class C 1 and of full rank at x , 
and let y° := F(x a ). Assume further that N2 is finite dimensional. Then there 
exists an open neighborhood V C N 2 of y and a map 

G:V^O 

of class C 1 such that G(y°) = x° and so that 

F(G(y)) = y 

for each y G V. □ 

This is an immediate consequence of the Rank Theorem for finitedimensional 
mappings, which can be found in any advanced calculus book, because if F has 
full rank, then there is some finite dimensional subspace 

TVi C Ni 

so that the restriction of F to 

still has full rank, and the finite dimensional theorem can be applied to the 
restricted mapping. 
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The Implicit Function Theorem follows from this. In general, if 

F : O x Q -> N 3 

is differentiable at 

{x°,y°) eOx QCN! xN 2l 

where Ni, N 2l N 3 are normed spaces, the partial derivative with respect to N 2 
at (x ,y ) is the linear bounded operator 

^-F[x°,y°}:N 2 ^N 3 
dy 

given by 

F.[x°,y°](0,-), 

which can be identified naturally to the differential of 

F(x ,-) 

as a map from Q into N 3 . If this partial derivative is of full rank, then also the 
mapping 

F:Ox Q^iVx xN 3 :(x,y)^(x,F(x,y)) 

is of full rank. Applying the Rank Theorem to F we conclude the Implicit 
Function Theorem: 

Theorem 53 Assume that F : O x Q —> N 3 is of class C , where O, Q are open 
subsets of normed spaces N\ and N 2 , respectively. Assume further that -g-F is 
of full rank at (x , y ), that z° = F(x°,y°), and that both Ni and N3 are finite 
dimensional. Then there exist open neighborhoods V C JV3 0/ z° and W C O of 
x° and a map 

G:W x V -» Q 

of class C 1 such that G(x°, z ) = y° and so that 

F(x, G(x, z)) = z 

for each {x, z) G W x V. □ 

The following Lemma is particularly useful when applied to mappings send- 
ing controls into final states. In those cases one often applies the result to 
conclude under certain conditions that controllability using arbitrary controls 
implies controllability with nice controls, such as piecewise constant controls or 
polynomial ones. 

Lemma B.2.2 Assume that N\ is a dense subspace of N\, that / : O — » N 2 is 
of class C , and that dimiV2 = n < 00. Assume further that / has full rank at 
some x° € O. Then there are elements x° in 



C:=Op|A^i 
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arbitrarily close to x° such that the mapping 

f:=f\ d : O^N 2 

is of full rank at a; . If x° € O, then / is of full rank at x°. 

Proof. Note that / is also of class C , directly from the definition of differen- 
tiability; its differential at any x £ O is the restriction of /* [x] to Ni . Let 

Vi,...,V n 

be elements of JVi so that 

Mx°}(vi),...J4x°}(Vn) 

form a basis of N 2 . Since f*[x ] is continuous, we may approximate the Vi's by 
elements of Ni while preserving linear independence. So we assume from now 
on that Vi G N\ for all i. If x a G N\, then this means that / has full rank there, 
as wanted. If not, consider a sequence 

Xj — ► x as i^oo 

of elements of O. By continuous differentiability, 

lim f*[xj]{vi) = f*[x]{vi) 

for each i. It follows that for all j large enough 

f*[xj](vi),...,f*[xj](v n ) 
are linearly independent, and they are the same as 

f*[Xj\{vi ),..., f*[Xj\{v n ) 
because the Xj's are in N 2 - I 



Appendix C 

Ordinary Differential 
Equations 



As explained in the text, it is convenient to introduce some concepts from 
Lebesgue integration theory. However, readers who have not seen measure the- 
ory before still should be able to follow the rest of this book if they substitute 
"piecewise continuous function" instead of "measurable function" and they in- 
terpret "almost everywhere" as "everywhere except at most for a countable set 
of points." At some points in a few proofs, results from Lebesgue integration 
theory will be used, but all statements as well as most proofs will not require such 
knowledge. Two good references for real- variables concepts are [190] and [264]. 
Except for some notation introduced there, the next section can be skipped by 
readers willing to restrict themselves to piecewise continuous controls. 

C.l Review of Lebesgue Measure Theory 

The simplest way to introduce measurable functions is by first defining sets of 
measure zero and then defining measurable functions as the pointwise limits of 
step (piecewise constant) functions. We do this next. 

A subset S of R™ is said to have zero measure if for each e > there is a 
countable union of balls B\ , Bi , ■ ■ ■ of volume £; such that 

oo 

?=1 
and 

oo 

S C |J B t . 
Two functions f,g:0—>T from some subset O C 1" into some other set T 
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are said to be equal almost everywhere (a.e.) if the set 

{xeO\f(x)^g(x)} 

has zero measure. In general, a property is said to hold almost everywhere 
provided that it only fails in a set of measure zero. 

If IX is a metric space and {fi,i = 1,2,...} is a sequence of functions from 
O C W 1 into IX, and also / : O — > IX, one says that {fi} converges to f almost 
everywhere, denoted 

fi -*• / (a.e.) 

if the set 

{x e O | fi(x) -h /(*)} 

has measure zero. In other words, except for a negligible set, the sequence 
converges at every point. (For instance, 

fi(x) : [0, 1] - [0, 1] : a; .-> a;' 

converges almost everywhere to the function f(x) = 0.) 

Let X be an interval in R and IX a metric space. A piecewise constant function 
g : T — » IX is one that is constant in each element Ij of a finite partition of X 
into subintervals. An / : O — > IX is a measurable function if there exists some 
sequence of piecewise constant functions {/j, i = 1, 2, . . .} so that {fi} converges 
to / almost everywhere. Clearly, continuous functions are measurable, and in 
general g o / is measurable if / is measurable and g is continuous. 

By abuse of notation, it is common to identify functions that are equal almost 
everywhere. In particular, given aw£ W-' T < T > we may, and will, also think of it 
as a function defined on the closed interval [a, t] . 

The next two Remarks do use some less trivial facts from Lebesgue measure. 
However, they are not needed for the understanding of the further material in 
this Appendix, and only play a role in an approximation result in the main text. 

Remark C.l.l Assume that the metric space II is separable, that is, it has a 
countable dense subset. Consider the following property for an u> : T —> IX: 

io~ (V) is measurable for each open subset V C IX , (C.l) 

where "measurable" means Lebesgue measurable. (The measurable sets are 
those obtained by countable intersections, unions, and complements, starting 
with open subsets of X and sets of measure zero.) Then, the map lu is measurable 
if and only if it satisfies Property (C.l). This is a very useful fact, standard 
when IX is a subset of a Euclidean space. In this generality, it is also known but 
less accessible, so we give now a proof. The necessity of (C.l) is a consequence 
of the fact that an almost everywhere limit of functions satisfying (C.l) again 
satisfies (C.l) (see, for instance, [264], Fact M7 in Chapter X), and the fact 
that piecewise constant functions obviously satisfy (C.l). The converse can be 
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proved as follows. Assume that u> satisfies the property. By [264], Fact Mil, 
and Remark 3 on page 234 of that reference, uj is known to be the limit of 
simple functions. Recall that lu is simple if there is a partition of I into finitely 
many measurable sets Jo, . . . , Jk such that J\, . . . , </& have finite measure and 
lu is constant on each Jj. It will be enough, then, to show that every simple 
function is an almost everywhere limit of piecewise constant ones. So let lu be 
simple, and assume that 

oj(t) = Ui if t is in J, 

and that all the Uj's are distinct. 

We claim that it is enough to consider the case in which k = 1. Indeed, 
assume that one has found sequences {oJn , n = 1,2,...} and sets Zi of measure 
zero for each i = 1, . . . , k, such that for each i and each t £ Zi there is some 
N = N{t, i) such that 

for all n > N (intuitively, these are approximating the "characteristic functions" 
of the Jj's, but with values U{, u instead of 1, 0). Then, we can define a function 
uu n as follows: For any t € I, if there is exactly one of the w„ '(t) equal to u, 
and the rest equal uq, then we let w„(t) := u,; otherwise, we let to n (t) := uq. 
This is again piecewise constant, as it is constant on each interval where all of 
the OJn S are constant. Now, for each 

t i Z := Zi U . . . U Z k 

there is some N so that u> n (t) = u)(t) for all n > N: It is enough for this to take 
N > N(t, i) for this t and i = 1, . . . , k. So we assume that k = 1. 

The problem is now that of showing that if J C T has finite measure and u, v 
are two elements of IX, with U)(t) = u if t G J and u>(t) = v otherwise, then there 
exists a sequence of piecewise constant functions {io n } so that u>„(t) = ui(t) for 
all n = n(t) large enough, except on a set of measure zero. To see this, start 
with a sequence of open sets V n so that 

n(V n AJ) -» , 

where /! indicates Lebesgue measure and A indicates symmetric difference. Since 
each V n is a countable union of intervals and has finite measure, without loss 
of generality we can assume that each V n is a finite union of (open) intervals. 
Furthermore, we may also assume that 

n(V n AJ) < 2-" 

for all n. Let u> n be equal to u on V n and v otherwise; this is a piecewise constant 
function. It agrees with lu whenever t ^ V n A J. Letting 



W N := |J (V n AJ) 



2>N 
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one has that h(Wn) — > 0, so also Z := (~}{Wn,N > 0} has zero measure, and 
for each t ^ Z one has that u) n (t) = u)(t) for large N. □ 

A function uj : X — > U is (essentially) bounded if it is measurable and there 
exists a compact subset ifCU such that 

uj(t) € -K" for almost all t & X , 

and it is locally (essentially) bounded if the restriction of uj to every bounded 
subinterval of X is bounded. 

Remark C.1.2 Assume, as in Remark C.l.l, that IX is separable. Then: If uj is 
essentially bounded, then there is always a sequence of piecewise constant func- 
tions cu n converging almost everywhere to uj and equibounded (that is, 0J n (t) € K 
for almost all t, for some compact K independent of n). This is because if K 
is a compact containing the values of uj, we may replace w by a function al- 
most everywhere equal having image entirely contained in K, and now see K 
itself as a separable metric space (as a subspace of 11). Then Property (C.l) 
holds for maps into K, so uj is measurable also as a map into K; therefore an 
approximation can be found with values entirely contained in K . 

Often one can obtain approximations by more regular controls. For instance, 
assume that 11 is a convex subset of K™. Then each piecewise constant control 
can be approximated almost everywhere by continuous controls (just interpo- 
late linearly on small intervals about the discontinuities), and any bound on the 
norm of the original control is preserved. Thus, every essentially bounded mea- 
surable control can be approximated in this case by an equibounded sequence 
of continuous controls. If in addition U is open, then one can approximate, as 
long as the interval X of definition is finite, by analytic, and even polynomial, 
controls. This is because one can first assume that the interval is closed, then 
approximate by continuous controls, and then approximate these uniformly via 
the Weierstrass Theorem by polynomials in each coordinate; openness of 11 
guarantees that the values of the controls eventually lie entirely in II. □ 

For measurable functions h : X — > C" on a finite interval one may define the 
Lebesgue integral J h(r)dT via the limits of integrals of suitable sequences of 
approximating piecewise constant functions. (For essentially bounded functions, 
the only ones that are really used in this text, any bounded sequence of piecewise 
constant functions will do.) One can also define integrals of functions on infinite 
intervals in a similar fashion. An integrable function is one for which J \\h(r)\\ dr 
is finite; the function h is locally integrable if 

b 

\\h(r)\\ rfr < oo 

J a 

for each a < b, a, 6 G T. 
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A function £ : X — > C n defined on a compact interval X = [a, b] is said to be 
absolutely continuous if it satisfies the following property: for each e > there 
is a 5 > such that, for every k and for every sequence of points 

a < ai < bi < a,2 < bi < . . . < cik < bk < b 

so that y] ■ 1 (b ; ; — dj) < <5, it holds that y]._ 1 ||£(6,) — C( a i)ll < £ - The function 
£ is absolutely continuous if and only if it is an indefinite integral, that is, there 
is some integrable function h such that 

^(t) = £(a) + [ h(r)dT (C.2) 

<J a 

for all t € X. An absolutely continuous function is differentiable almost every- 
where, and 

&) = h(t) 
holds for almost all t, if h is as in (C.2). Note that a function is Lipschitz con- 
tinuous if and only if it is absolutely continuous and has an essentially bounded 
derivative. If X is an arbitrary, not necessarily compact, interval, a locally abso- 
lutely continuous function £ : X — > C™ is one whose restriction to each compact 
interval is absolutely continuous; equivalently, there is some a £ X and some 
locally integrable function h so that £(£) = £(er) + J h(r) dr for all t € X. 

If X is an interval and IX is a metric space, we denote the sets of all (essen- 
tially) bounded measurable maps and all locally bounded maps into II by 

C%(1) and L^ loc {X), 

respectively. When IX = R™ (or C", depending on context) we write simply 

£~(2) and L™' loc (X) , 

respectively. Elements of such spaces are equivalence classes of functions that 
coincide almost everywhere, and there is no distinction between 

£%[v,t), £™[o-,t], £%{<t,t), £™(o-,t] 

since these can be all naturally identified with each other. The space £^°(Z) is 
a Banach space under the norm 

HeiU := ess.sup. {||e(*)ll ,t € X}; 

that is, H^Hoo is the infimum of the numbers c such that {t \ ||£(i)|| > c} has 
measure zero, and with the distance 

d(£,C) = ll£-ClL 

the space is a complete metric space. This is the standard "sup norm" save 
for the fact that for measurable functions one ignores values that occur only 
exceptionally. 
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More generally, for every metric space IX we view C^(T) as a metric space 
(itself complete, if IX is complete) when endowed with the distance 

doo {u,v) := sup d(w (<),!>(<)) 

tei 

(uniform convergence), again understood in the "essential" sense that behavior 
on sets of measure zero is disregarded in computing the supremum. 

If a < t are finite, the notation C„(<7, r) is used for the subspace of ££°(cr, r) 
consisting of all continuous maps 

£:[a,T]-»R n . 

(This is of course very different from the space of continuous maps on the open 
interval; it may be identified to the space of such maps whose one-sided limits 
exist at t — > a + and t — > t~ .) This is again a Banach space. It follows that if p 
is any positive real number, the ball centered at any £° £ C„(X), 

B p (£°) := {C € C° n (T) | He-Clloo <P>. (C3) 

is also a complete metric space. We shall use the fact that the Contraction 
Mapping Theorem holds on each such ball as well as on the whole space. That 

is, if 

S : B p (?) -» B P (£°) 

is any map such that, for some A < 1, 

H^-sciL^AHe-ciL 

for all £, £, then S has a (unique) fixed point in B p (£ ), and similarly for the 
entire space (p = oo). 

Remark C.1.3 Assume that X is an interval and IX is a metric space. Suppose 
that we have a convergent sequence u% — > u> in £^(1). We prove here that the 
set of these controls is equibounded, that is, there is some compact subset K of 
IX so that, for almost all t £ I, u>(t) £ K and u>i(t) £ K for all %. By definition 
of £??(!) there are compact subsets K and Ki, i > 1, so that, except for t £ X 
in a set of zero measure, u)(t) £ K and u>i(t) £ Ki for all i. For each e > 0, let 
B e be the set of points at distance at most e from K . Since LUi — > oj in uniform 
norm, for almost all i we have that d (o>,(£), U>(t)) < £,, where {e^} is a sequence 
of real numbers converging to zero, and so u>i(t) £ B £ . for all i. (Observe that 
these sets need not be compact unless, for example, one assumes the space IX 
to be locally compact.) Replacing Ki by its intersection with B 6i , we may take 
each Ki as contained in B 6i . It will be enough to show that the set K defined 
as the union of K and of all the sets Ki is also compact. We pick a sequence 
{ui} of elements of K and show that it admits a convergent subsequence in K . 
If this sequence is contained in a finite union A = K\ (J K 2 U ■ • ■ U ^r > then 
it has a convergent subsequence, since A is compact, and we are done. So 
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assume that the sequence is not contained in any finite union of this form. 
Then, there are subsequences Uj t and Kk t so that Uj e € Kk e for all £. (Such 
a subsequence can be obtained by induction: if we already have j\ , . . . , jg and 
ki, ■ ■ ■ , kg, let A = K\ (J Ki U • • • U Kk t an( i ^ B be a union of finitely many 
fQ's which contains ui, ...Uj t ; as the entire sequence cannot be contained in the 
union of A and B, there are j > jg and k > kg so that Uj £ K^, so we may 
take jg+i := j and kg + \ := k.) We renumber and suppose from now on that 
Ui € Ki for each i. Pick for each i = 1, 2, . . . some r/i G K so that d (uj, j/j) < £j. 
Since .K" is compact, there is a subsequence of 3/1, 2/2 ; • • • which converges to some 
element y G K, and renumbering again, we assume without loss of generality 
that yi — > y as i — > 00. Thus Uj — > y as well, so the original sequence has a 
convergent subsequence, establishing compactness. □ 

C.2 Initial- Value Problems 

When dealing with continuous-time systems, it is necessary to have a good 
understanding of the basic facts regarding initial- value problems 

£ = f(t,S(t)), ^°) = A (ivp) 

where £(£) G X C W 1 . Such equations result when a control is substituted in 
the right-hand side of a control equation 

x = f{x,u). 

(As a rule, when we are proving results for differential equations we use £ to 
denote a solution and reserve the letter "x" for states, that is, values £,{t). 
During informal discussions, examples, or exercises, we often revert to the more 
standard use of x for the solution function £ itself. Similar remarks apply to the 
notation lu for control functions and u for control values.) 

Since it is essential to allow for the possibility of discontinuous controls, 
the right hand side of (IVP) cannot be assumed to be continuous on its first 
argument t, and hence this results in a possibly nondifferentiable solution £(•). 
Thus, the meaning of "solution" of the differential equation must be clarified. 
The simplest approach is to define a solution of (IVP) on an interval X as a 
locally absolutely continuous function 

i-.X^I (C.4) 

such that the integral equation 

£(t) = z°+ / f{T^{r))dT (INT) 

holds. In order for this integral to be meaningful, we first remark that f(t,^(t)) 
is measurable. More precisely, we assume that 

/:Ixl^R" 
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is a vector function, where X is an interval in M. and X is an open subset of R", 
and that the following two properties hold: 

f(-,x) : I — * W 1 is measurable for each fixed x (HI) 

and 

f(t, ■) : M" -» R™ is continuous for each fixed t . (H2) 

Under these hypotheses, it is also true that 

is measurable as a function of i, for any given continuous (C.4). This is proved 
as follows. If £ would be piecewise constant on a partition I\ , . . . , /& of I, say 
with £(£) = Xi when t € ij, then 

fe 
/(*,£(*)) =£/(t,*i)Xi(t), 

where Xi is the characteristic function of 1^. This function is a sum of products 
of measurable ones, hence measurable itself. If now £ is continuous, we can 
always find a sequence of piecewise constant functions {£j} such that 

&-£ 

(uniformly on compact subsets of X). Since / is continuous on the second 
coordinate (assumption (H2)) it follows that pointwise 

/(*,&(*))-/(*,£(*)), 

and therefore the latter is also measurable as claimed. 

Definition C.2.1 A solution of (IVP) on an interval J CI containing <r° is 
a locally absolutely continuous function £ : J — » X suc/i i/iat (INT) ZioWs /or aW 
t6J. □ 

In most cases in this text, the derivative of £ is (almost everywhere) bounded 
on each finite interval, because the right-hand side of equations involve controls 
(which are assumed, by definition, to be essentially bounded) and the continuous 
function £. Thus, solutions will be in fact locally Lipschitz, not merely absolutely 
continuous. 



C.3 Existence and Uniqueness Theorem 

The following result is known as the Bellman- Gronwall or the Gronwall Lemma. 
It is central to the proof of uniqueness and well-posedness of ODEs. By "inter- 
val" we mean either finite or infinite interval. 
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Lemma C.3.1 Assume given an interval T C R, a constant c > 0, and two 
functions 

a,n : T — > R+ 

such that a is locally integrable and /! is continuous. Suppose further that for 
some a G I it holds that 

H(t) < v(t) :=c+ / a(r)^(T)dr (C.5) 

for all £ > cr, £ € 1. Then, it must hold that 

fi(t) < ce^ a(T) dT . (C.6) 

Proof. Note that v(t) = a(t)/j,(t) < a(t)v(t) almost everywhere, so 

u(t) - a(t)u(t) < (C.7) 

for almost all t. Let 

7r(£) :=v(t)e-ti a ( T)dT . 

This is a locally absolutely continuous function, and 

7r(t) = [i>(t) - a(t)v(t)] e~ & " (r) dT 

is < by (C.7). So n is nonincreasing, and thus, 

v(t)e-5>^ dT = Tr(t) < tt(ct) = v(a) = c, 

from which the conclusion follows. I 

The following remark is very simple, but the resulting inequality will appear 
repeatedly. 

Lemma C.3.2 Let f,g : L x X -> R" both satisfy the hypotheses (HI), (H2), 
where L is an interval in R, let Xq C X C R™ be two subsets, and let 

£ : L -> A" and C = £ ~> ^o 
be continuous. Suppose that there exist two locally integrable functions 

a,/3: L^R + 

such that 

||/(i,aO-/(i,|/)||<a(t) ||ar -s/ll (C.8) 

for all x, y € X and all t€i, and 

H/M-sCM) II </?(*) (C9) 
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for each x £ Xq and all t £ L. Then, if x , z° are arbitrary elements of K n and 
<t° G L, and we denote 

e 



£(*):= x + / /(t,£(t))oY 



and 



C(t):=z + / g(T 7 ((T))dT, 

J a 

it holds that 

tit) - C(i) < ||:r - z°|| + /" a(r) U(r) - C(r)|| dr + f /3(r) dr 

for all t > a such that t € L. 

Proof. Just write 

f(r, - g(T, C) = f(r, - /(r, + /(r, C) - g(r, C) , 

take norms, and use the triangle inequality. I 

We now state the main Theorem on existence and uniqueness. 

Theorem 54 Assume that f : X x X — » K™ satisfies the assumptions (HI) and 
(H2), where X C R™ is open and X C R is an interval, and the following two 
conditions also hold: 

1. f is locally Lipschitz on x; that is, there are for each x° € X a real number 
p > and a locally integrable function 

a:X^R + 

such that the ball B p (x°) of radius p centered at x° is contained in X and 

\\f(t,x)-f(t,y)\\<a(t)\\x-y\\ 

for each t € X and x,y £ B p (x°). 

2. f is locally integrable on t; that is, for each fixed x° there is a locally 
integrable function (3 : X — » R + such that 

|| /(*, *°) || </?(*) 

/or almost all t. 

Then, for each pair (a ,x ) 6lxl i/iere is some nonempty subinterval J C.X 
open relative to X and there exists a solution £ of (IVP) on J , with the following 
property: If 

C : J' -» X 
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is any other solution of (IVP), where J' C X, then necessarily 

J' C J and C = C on J' . 

The solution £ is called the maximal solution of the initial-value problem in 
the interval X. 

Before proving the Theorem, we remark that the assumptions imply also a 
stronger version of Condition 2, namely that it holds also uniformly on compacts. 
That is, for any compact K C X there is a locally integrable 7 such that 

\\f{t,x)\\ <7(t) for all teX,xeK. (CIO) 

This is proved as follows. Given any x° G K , there are p > and functions a, (3 
as in the Theorem. Thus, for any element x £ B p (x°) and each t G X, 

\\f(t,x)\\ < \\f(t,x )\\ + \\f(t,x)-f(t,x°)\\ 

< 0(t) + pa(t) ■ ( c - n ) 



Call this last function 7 x o and note that it is locally integrable, too. Consider 
the open covering of K by the sets of the form B p (x°); by compactness there is 
a finite subcover, corresponding to, say, balls centered at x\, . . . ,xi. Take then 

7(£) :=max{7 a:i ,...,7 Xl }. 



This is again locally integrable, and (C.ll) gives then the desired Property 
(CIO). □ 

We now prove the Theorem. 
Proof. Assume without loss of generality that X 7^ {cr }; otherwise the 
problem is trivial. We first show that there exists some 5 > such that problem 
(IVP) has a solution on the interval [<j , o~ + 5} f]X. If a is the right endpoint 
of X, then this is obvious, so we assume that such is not the case. Pick, for the 
given x° , a number p and locally integrable functions a and (3 as in the statement 
of the Theorem. Note that, because of local integrability, the function 



r-t+a" 

a(t) := I a{ T )dT -> 



as t — > + , and similarly for 

b(t):= / (3{r)dT. 

J a" 

Note also that both a and b are nonnegative and nondecreasing. Thus, there 
exists a S > such that a + S € X and 

(i) a(t) < a{5) = X < 1 for all t € [0, 5], and 
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(ii) a{t)p + b{t) <a{5)p + b{6) < p for all t€[0,S\. 

Let £o = x° on the interval [a , <r° + 6] and consider the ball of radius p centered 
at this point in the space C®(a , a a + S), 



Introduce the operator 
defined by 



B ■■= B P (C ) . 
S:B->C°(<j ,<t° + 5) 



(50(i):=x°+ / f(T,ar))dr 



Since the values ^(t)^ G [cr ,cr° + 6] all belong to some compact set K (by 
continuity of £,) it follows from the discussion before the proof that /(•, £(•)) is 
locally integrable, since 

||/(r,e(r))|| dr < f 7 (r) dr . 

o J a a 

Thus, S is well defined, and S*£ is absolutely continuous, for each £. We claim 
that B is invariant under S. Indeed, take any £ G B and apply Lemma C.3.2 
withL= [a°,S + a°],g = 0, same a, /3, a: , £,/ as here, X = B p {x°), X = {x }, 
C := £o> an d z ° = x °- Since then ^ = S£ and C = Co, we conclude that 

\\S^-^\\ co <U-^\\ cc a(S) + b(5)<p, 

the last inequality because of Property (ii). Therefore, S can be thought of as 
a map B — ► B. Next we prove that it is a contraction there. 

For that we again apply Lemma C.3.2, this time with the following choices, 
given any ^,( e 6: X = X n = B p (x°), a,f,x° as here, g = /, /3 = 0, and 
L = [a , <5 + fi ]. Note that now the first and third terms in the conclusion of 
the Lemma vanish, and we have 

l|SS-SCIIoo<A|K-Clloo> 

so S is a contraction. 

It follows then from the Contraction Mapping Theorem that there is some 
fixed point £ = S£, and this is, then, a solution of (IVP) in the interval [fi , S + 
a }. If a is not a left endpoint of T, a similar argument proves that there is a 
solution in some interval of the form [a — 5, a ] , and therefore by concatenating 
both solutions, we conclude that there is a solution in an interval [er° — 6, <r° + 5} . 
If <7 is an endpoint, we have a solution in an interval of the form [a , a + 5} or 
[a — 5, a } , and in any case this gives a solution defined on a neighborhood of 
<T° in X. Moreover, this local existence result holds for every initial pair [a , x a ). 

We now prove the following uniqueness statement: If £ and £ are two solu- 
tions of the problem, (IVP) on some interval J C X, cr° G J , then £ = C, on J . 
We prove that they coincide for t G J, t > a , the case t < a being analogous. 
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We first prove that £ and ( must coincide in some (possibly small) interval 
of the form [a a ,a° + S], for some 5 > 0. (If er° is the right endpoint, there 
is nothing to prove.) Choose again p as in Condition 1, for the given x Q . By 
continuity of both £ and £, there is a 5 small enough that all values £(£) and 
C(t), for £ € [<t , cr + 5], belong to the ball B p (x°). Once more we apply Lemma 
C.3.2, this time taking X = Xq = X, /, x°,£, £, a as here, z° = x°, g = /, 
L = [cr°,cr + 5], and /3 = 0. As before, the first and last term vanish, so the 
Lemma gives that 

U(t)-C(t)\\< f a(T)U(r)-C(r)\\dT 

for all t € [a ,a+5]. GronwalPs Lemma then gives that the left-hand side 
must be zero, since "c" is zero. This proves uniqueness on that (small) interval. 
Assume now that there would exist any t > a , t € J, such that £(t) ^ (,{£)■ 
Define 

a 1 := inf{£ £ J,t > a \ £(t) ^ ((t)} . 

So £ = ^ on [cr ,?! 1 ), and hence by continuity of both £ and C it also holds 
that C(°' 1 ) = C( ' 1 )- The above local uniqueness proof was equally valid for any 
initial a G X and any x° € X. Thus, we apply it to the initial-value problem 
with a 1 as initial time and initial state x 1 := ^(ct 1 ). It follows that £ = C, on 
some interval of the form [cr 1 , cr 1 + 5] for some 6 > 0, contradicting the definition 
ofo- 1 . 

We now show that there is a maximal solution. For this, consider 

Tmin '■= inf{£ g I | there is a solution on [t, a ]} 

and 

Tmax := sup{£ G I | there is a solution on [a , t]} . 

(Possibly T max = +oo or T m ; n = — oo.) From the local existence result we know 
that r m i n < T max . Consider the open interval (T m i n ,r max ). There is a solution 
defined on this interval, because there are two sequences 

^n 1 7"min anu T n \ T max , 



; max- 



and there is a solution on each interval (s n ,t n ), by definition of T m - ln and r„ 
and these solutions coincide on their common domains by the uniqueness state- 
ment. 

We now pick J as follows. If r m j n and r max are both in the interior of I, 
then 

J := (t„ 



' mm 5 ' max; 



If Tmin is the left endpoint (that is, X has the form [r m i n ,6] or [r m i n ,6)), then 
r min is added to J provided that a solution exists on an interval including T m - ln . 
Similarly for r max . With this definition, J is open relative to X and nonempty. 
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Moreover, if T m ; n is interior to 1, then there can be no solution Q of (IVP) 
which is defined on [r m i n ,cr ]. Otherwise, from the local existence Theorem 
applied to the initial-value problem with £(r m i n ) = C( r min), there would result 
a solution on some interval 

\T min ^i ^"minj 

and hence by concatenation also one on (r m i„— d, a ], contradicting the definition 
of T m j n . Similarly for T max . In conclusion, any solution of (IVP) must have 
domain included in the above J, and therefore the last part of the Theorem 
follows again from the uniqueness statement. I 

When T max is in the interior of X, one calls t = T max a finite escape time for 
the problem (IVP). 

Exercise C.3.3 Let X = I = R. Consider the differential equation 

x = x 2 

with the following initial values, and find explicitly the maximal solution in each 
case: 

1. x(0) = 1. 

2. x{\) = 0. 

3. x(l) = 1. □ 

The local Lipschitz property typically is not hard to verify, due to the fact 
to be proved next. 

Proposition C.3.4 Assume that / : I x X — > R" satisfies assumption (HI), 
where X is open in R" and X is an interval. Suppose further that for each t the 
function fit, ■) is of class C 1 and that for each compact set K C X there is some 
locally integrable function a : T — > M + such that 



df , ^ 

ix^ 



<a(t) for all t € X, x € K . (C.12) 



Then / satisfies the local Lipschitz Property (1) in Theorem 54. In the particular 
case when / is independent of t, Property (C.12) is automatically satisfied. 

Proof. Given any x° € X, pick any p such that B p (x°) C X. Apply the above 
property with K := B p (x°). Thus, for any two elements x, y in K we have that 
for each coordinate fi, i = 1, . . . , n of /, 

fi(t,x) - fi(t,y) = ^2 I -Q^{ t ^ z ){ x ] -Vj) 
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for some z in the line joining x and y, by the Mean Value Theorem applied to 
the function fi(fix + (1 — fi)y) of /i G [0, 1]. It follows that for some constant c, 

||/(t,aO-/(t,j,)||<ca(i)||a;-y|| , 

so ca(t) is as desired. If/ (or even if just df/dx) is independent oft, continuity 
of df/dx insures that this is bounded on compact sets. I 

The proof of the Theorem shows that solutions exist at least on the interval 
[<7 , <7 +6], if S is so that properties (i) and (ii) in the proof hold. This gives 
(very conservative) lower bounds on the interval of definition. The next problem 
illustrates how one may use such estimates, for an equation that arises naturally 
from a control system. 

Exercise C.3.5 Consider the system 

x(t) = x 2 {t) + u(t) . (C.13) 

Show that there is a solution with x(0) = 1, defined on the interval [0,0.1], if u 
is any control that satisfies 

o.i 

\u(r)\dT < 0.1. 
/o 

(Hint: Try p := 1 in the Theorem.) □ 

The following fact is often useful. 

Proposition C.3.6 Assume that the hypotheses of the above Theorem hold 
and that in addition it is known that there is a compact subset K C X such 
that the maximal solution £ of (IVP) satisfies £(£) G K for all t G J. Then 

J=[a° 7 +oo)f)T 7 (C.14) 

that is, the solution is defined for all times t > a , t Si. 

Proof. Assume that (C.14) does not hold. Thus, £ is defined on [er°,T max ), 
r max < oo, but there is no solution on [er°,T max ], and T max G T. Find a 7 as in 
the discussion preceding the proof of Theorem 54, for the compact set K in the 
statement of the Proposition, and consider the function 

en\.-i £(*) ifie [*Vmax) 

SI K l ) ■- S * jf f _ 

^ ^ IJ - L — ' max 

where x* is any element in K. Thus, f(t,£(t)) is bounded by "f(t), which is an 
integrable function on [er°,T max ], and is therefore integrable itself. Define now 

£(w):=z°+ r°'" /(r,f(r))dr. 
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Since this is the limit of the integrals J as t — » T max and K is closed, £(r max ) G 

K and in particular £(r max ) € X. Also, since /(r, £(t)) = /(t^C - ?")) almost 
everywhere, we conclude that £ satisfies 

£(*) = x°+ [ /(r,e(r))dT 



= x°+ f f(r,ar))dr 

on the interval [<r°,r max ]; thus, it is absolutely continuous (from the first equal- 
ity) and it satisfies (IVP) (from the second). So we defined an extension to 
[cr°,T max ], contradicting maximality. I 

A related result is as follows: 

Proposition C.3.7 Assume that X = R ra and that the hypotheses in Theorem 
54 hold but with the boundedness Property (2) being global: There is some 
locally integrable (3 such that 

|| f(t, x) || </3{t) for all x el". 

Then again (C.14) holds. 

Proof. We use the same argument as in the proof of Proposition C.3.6, except 
that now /(£,£(£)) is bounded by (3 instead of 7. I 

Note that the assumption that X = R™ is crucial in having a well defined 
C( r max) m this argument. Otherwise boundedness is not sufficient, as illustrated 

by 

£ = 1, x ° = 

on X = (-1, 1) and I = R, for which J = (-1, 1) ^ 1. 

Finally, another sufficient condition for global solutions to exist is that the 
Lipschitz condition holds globally: 

Proposition C.3.8 Let X = R™ and assume that [it , +00) C X. Suppose that 
/ satisfies the assumptions of Theorem 54 except that the function a can be 
chosen independently of x , with p = 00; that is, that there exists a locally 
integrable a so that 

||/(t,a;)-/(t ) y)||<a(t)||a:-y|| 

for each t G X and x, y G R™. Then J = [a , +00). 

Proof. This is proved in a manner very similar to Theorem 54. Let S be such 
that (i) in the proof of the Theorem holds, i.e., so that 

a(t) < a{5) = X < 1 for all t G [0, 6] . 
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Then, the proof of the Theorem, arguing with all of C^(cr ,a a + 5) instead of 
Bp(^a), shows that solutions exist for (IVP) at least on the interval [a , cr° + 5], 
and that this is true for any such initial- value problem, independently of the 
particular a and x Q . Given now a fixed a; and it , consider the maximal 
solution £. Assume the Proposition to be false, so in particular r max < oo. Let 
&i '■= T ma x — 5/2 and x\ := £(ci). The initial- value problem has a maximal 
solution £ with these initial data and, as remarked before, C, is defined at least 
in the interval 

[<7l,<7l + 5} . 

If we let £(t) be defined as £(£) for t < o\ and equal to £(i) for t > <7i, we obtain 
a solution of the original (IVP) on 

[a°,r max + <5/2], 

contradicting maximality of T max . I 

Exercise C.3.9 Consider the nonlinear pendulum 

X\ = X 2 

&2 = — sin^i + u{t) 
with 1 = R. 

1. Prove that if u € C^' loc {K) and a°,x are arbitrary, the corresponding 
initial- value problem has r max = +oo. 

2. Prove the same fact for the equation that results when one considers the 
closed-loop system resulting from the feedback law u = —2x\ — X2- 

3. Find a feedback law (and an initial state) for which the closed-loop system 
has a finite T max . □ 

Exercise C.3.10 Show that with the feedback u = — x 3 , system (C.13) in Ex- 
ercise C.3.5, with X = X = R, has no escape times, for any initial condition. □ 

All of the facts given above hold for complex initial-value problems (IVP) 
for which the set X is a subset of a complex Euclidean space C™ and / takes 
values in C". It is only necessary to consider separately the resulting differential 
equations for the real and imaginary parts of £ and /. That is, the resulting 
equation is equivalent to an equation in R 2n , and the hypotheses of the various 
results must be verified for this real system. Observe that properties such as 
local integrability, essential boundedness, or continuity, can be defined in terms 
of complex numbers, using the norm 



N 
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in C™, where |A| is the magnitude of the complex number A. Equivalently, one 
may use the standard Euclidean norm in R 2n and ask that these properties hold 
for the corresponding real systems; the same concepts result. 

It is many times the case that / in (IVP) is smooth in both x and t; in control 
applications this will correspond to the case in which the dynamics as well as 
the control being applied are well behaved. In that case one may conclude that 
solutions are smooth as functions of time: 

Proposition C.3.11 Assume that /:Ixl^ R n is of class C k , where k > 1. 
Then, the conclusions of Theorem 54 hold, and the maximal solution £ of (IVP) 
is of class C +1 . 

Proof. By Proposition C.3.4, we know that the hypotheses of the Theorem 
hold, so there is indeed a solution. By induction, assume that £ is of class C l , 
with some I < k. (The case I = is clear by the definition of solution.) Then, 
in the equation 

the right-hand side is of class C l . Thus, the derivative of £ is I times continuously 
differentiable, from which it follows that £ is of class C l+l , establishing the 
induction step. I 

The result also holds for functions of class C w . Recall that an analytic (or 
more precisely, real analytic) function 

/: V->M, 

where V is an open subset of K m , is one with the property that for each v° € V 
there exists some p > and a series 

oo oo 

f( Vl ,...,v m )=J2--- E a ll ... tm (vi-v a i yi...(v m -v a m y™ (C.15) 

i 1= i m =0 



convergent for all v G B p (v ). A vector function / : V — > R" is analytic 
provided that each coordinate is. We also say that / is of class C" . When V is 
not necessarily open, we define analyticity by the requirement that there exists 
an extension to an analytic map in a neighborhood of V. 

Proposition C.3.12 Assume that / : I x X -> R n is of class C w . Then, the 
conclusions of Theorem 54 hold, and the maximal solution £ of (IVP) is of class 

Proof. Because analyticity is a local property, and using the fact that solutions 
are unique, it is enough to prove that there is an analytic solution defined in some 
interval (cr° — <5, a +6) about a a . (Even if a is an endpoint of T, by definition of 
analytic function there is an extension of / to some open neighborhood of Ix X, 



C.3. Existence and Uniqueness Theorem 485 

so we may assume that a is interior to I.) Adding if necessary a coordinate 
satisfying 

£ra+l = 1) Cn+l(o- ) = <T , 

we may assume without loss of generality that / is independent of t, that is, 
/ : X — » R n . Since the series (C.15) for / is convergent about x a , it also defines 
an extension to an analytic function g of complex variables (z\, . . . , z n ) defined 
in a neighborhood Z of x° in complex space C n . We shall show that there is a 
continuous function 

£:{|ct°-ct| <<5}^ZCC" 

defined on some complex disk of radius 6 > about a , such that 

£((t) = x + f f{l{z))dz if |a° - a\ < 5 , (C.16) 



where the integral is interpreted as a line integral along the line joining a to 
<x, such that £ is analytic on \a° — a\ < S (as a function of a complex variable). 
Then the restriction of £ to the real interval (a — 8, cr a + (5) will be as desired. 
In order to prove the existence of £ we introduce for each positive real 5 the 
space T?^(a°,5) consisting of all those 

£:{\<j°-(t\ <5}^C n 

that have the property that they are analytic in the interior of the disk and are 
continuous everywhere, thought of as a normed space with 

IKIloo : = SU P H^)ll • 

|cr°-<x|<<5 

This space is complete and is hence a Banach space, as proved in most functional 
analysis texts (see, for instance, [399]). It is naturally identified to a subspace of 
the Hardy space H°° consisting of all functions that are analytic in the interior 
of the disk and are bounded. (Completeness is an easy consequence of Morera's 
Theorem: Any Cauchy sequence has a continuous limit, by the completeness of 
the space of all continuous functions on the disk, and a uniform limit of analytic 
functions must be analytic, since along each closed path one has that 



= / f n dz -> / fdz 

and therefore the integral of the limit must be zero, too.) 
We also introduce for each p > the ball 

B p (So)CV»(a°,8) 

consisting of all elements with ||£ — £o|| 5= Pi where £o is the function constantly 
equal to x°, and introduce the operator S : B p (£o) — > i3 p (£o) defined by 

(SO(<t) :=x°+ I" f(£(z))dz. 
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As before, both p and 5 can be chosen small enough that this is a contraction; 
the result follows from the Contraction Mapping Theorem. I 

We now prove that solutions depend continuously on initial conditions as 
well as on the right-hand side of the equation. 

Theorem 55 Let a : X — » R+ be an integrable function, where X = [a, r] is 
a bounded closed interval in R, X an open subset o/R™, and D a positive real 
number. Suppose that f and h are two mappings IxX^ R™ which satisfy 
the hypotheses of Theorem 54 (namely (HI), (H2), and the local Lipschitz and 
integrability properties), and that £ : X — » X is a solution of 

to = mm 

such that the D -neighborhood of its range: 

K = {x | ||ar-£(*)|| < D for some te [a,r}} 
is included in X. Let 

H(t) := J h{s,£{sj)ds teX 

and B_ := sup te:z: ||i?(£)||. Assume that 

max{i£,||^)-z°||} < £ e " «*>* (C.17) 

and, with g := / + h, 

\\g(t,x)- g(t,z)\\ <a(t)\\x-z\\ for all x,z € X and t € X. (C.18) 

Then, the solution £ of the perturbed equation 

C = g(t,Q = f(t,Q + h(t,Q, CM = ^° (C19) 

exists on the entire interval [a, t], and is uniformly close to the original solution 
in the following sense: 

IK-Clloo ^ (||e(^)-^||+ ; ff)e /; - (s)ds . (C20) 

Proof. Let £ : J — ► R™ be the maximal solution of the initial value prob- 
lem (C.19), J C X. This solution exists because / and h, and therefore g, 
satisfy the hypotheses of the existence theorem. For any t G J we have: 

m-C(t) = ZW)-z Q + I [g(s,as))-g(s,C(s))}ds - I h(s,t(s))ds. 
Therefore, for each t G J: 

U(t)-C(t)\\ < U(v)-z°\\ + \\H(t)\\ + /"a( S )ll£(s)-CO0ll ds 
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and hence from Gronwall's inequality we conclude: 

U(t) - CWII < (||£(<0 - z°\\ + H) e/JaW*' . (C.21) 

In particular, this implies that ||£(£) — C(^)ll < ^j an d therefore £(£) € if. 
Thus the maximal solution Q is included in a compact subset of X, which by 
Proposition C.3.6 implies that J = X, and this means that Equation (C.21) 
holds for all t € X, which is what Equation (C.20) asserts. I 

Remark C.3.13 In Theorem 1 (p. 57), and also in Section 4.2 (starting on 
page 147), we establish several additional properties regarding joint continuity 
of £(t) on t and the initial condition. □ 

The following characterization of maximal solutions is often very useful. 

Exercise C.3.14 Show that if £ : [a , r ma x) -^ 1 is a maximal solution of 
x = f{t,x) then 

lim U(t)\\ = oo. (C.22) 

(Hint: Suppose that there would exist a sequence t n /r max so that £(£ n ) con- 
verges to some state z°; a solution £ exists on [r ma x — S, r max + 6] , for some 6 > 0, 
with C(^max) = z °', now use continuity on initial conditions to extend £.) □ 

C.4 Linear Differential Equations 

We now review some facts about the solutions of linear differential equations. 
We take here K = R or C. 

Assume that A(-) is an n x n matrix of locally (essentially) bounded (mea- 
surable) functions on an interval X with values in K. Consider the initial- value 
problem 

i(t) = A(t)S(t) , £(<t ) = z . (C.23) 

We may assume that X = M. simply by defining A[t) = outside X. Since the 
right-hand side is globally Lipschitz, Proposition C.3.8 insures that there is a 
(unique) solution for all t > a . Solving the equation for t < a is equivalent 
to solving the reversed-time equation £ = — A£, which is also globally Lipschitz. 
Thus, solutions are defined on (— oo,+oo). Observe that, in the complex case, 
the resulting system in R 2n is again linear, so the same conclusions apply for 
both real and complex equations. 

A convenient way to study all solutions of a linear equation, for all possible 
initial values simultaneously, is to introduce the matrix differential equation 

X(t) = A(t)X(t), X(a)=I, (C.24) 

with X(t) G K nxn for each igl. Since this can be seen as a linear differential 
equation over IK™ , it has a global solution on (—00,00). 
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We denote by $(r, a) the solution of (C.24) at time r, that is, X(t). This 
is defined for each real a and t; $ is the fundamental solution associated to A. 
Observe that the solution of (C.23) is then given by 

£(*) = <&(*, a )* - 
More generally, the solution of a nonhomogeneous equation 

i = A(t)Z + v, {(a ) = x° (C.25) 

on the interval [a , t], where v is any locally integrable n- vector of functions, is 
given by the variation of parameters formula 

£(t) = $(t,a°)x n + [ $>{t,s)v{s)ds (C.26) 

also in terms of the fundamental solution. This formula can be verified by 
derivation; it is well defined because continuity of $ implies that the integrand 
is locally integrable. 

A number of properties of the matrix function $(r, <r) can be proved as 
simple consequences of the uniqueness of solutions of ode's. Note that equation 
(C.24) says that 

g r =A{t)®(t,o-). 

Lemma/Exercise C.4.1 Prove that, for all a, r, [i € K: 

(a) $(t,t) = /. 

(b) $(r,<7) = $(T,n)®(fj,,a) . 

(c) *(t,v)- 1 = *(v,t) . 

(d) °*gll = -*( < r,T)A(T). 

(e) If A{t) = A is constant, then $(t,ct) = e {T ~^ A . 

(f) det $(r, a) = e^ ^acc A(r) ^ . n 

Sometimes the term fundamental solution is used for the matrix function 
of just one variable <&(£) := <&(£, 0), instead of <&. These two objects carry the 
same information, since $ can be recovered from $ via the formula $(£,er) = 
$(t)$(a) _1 . 

The exponential e that appears in (e) above is defined by the matrix series 

f M = I + t A+-A 2 + ...+ l A n + _ _ 

2 n! 
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This series is normally convergent, and 

\\e tA \\<y i_ pir = e^l . 

II II /_^l n \ 

n=0 

One method of computing the exponential is via Jordan forms — not at all 
suitable for accurate numerical computation, however — or more generally by 
finding a similar matrix for which the exponential is easy to compute. When 
all of the eigenvalues of A are distinct, the matrix can be diagonalized. That is, 
there is an invertible matrix T and a diagonal matrix 

A = diag(Ai,...,A n ) 

(the Xi being the eigenvalues of A) so that 

T~ X AT = A. 

Then also A = TA'T -1 for all /, and therefore 

e 

Consider as an example the equation 

1 
-1 

corresponding to a pendulum linearized about the position 9 = (normalizing 
constants so that m = g = 1). Since the eigenvalues ±« of A are distinct, we 
can diagonalize; computing eigenvectors gives: 

1 lWi 0\/l/2 -i/2 
i -jJU -i M 1/2 i/2 



1 _ Te A T -i = Tdiag(e Al ,...,e A ")T- 



Thus 



e tA 



1 1 We" \ /l/2 -i/2 
i -i) \ e- u ) ^ 1/2 i/2 

(e it + e~ lt )/2 (e lt -e~ lt )/2i\ _ ( cost sint 
■(e 4 * - e- lt )/2i (e u + e~ u )/2 ) ~ \ - sini cost 

In the nonconstant case, the exponential is generalized to the use of a power 
series expansion, for 3>(t,cr), called the Peano- Baker formula, for any fixed real 
numbers a, t: 

$(t,a) = 1+ A{s 1 )ds 1 + / A(s 1 )A(s 2 )ds 2 ds 1 + ... 



t rsi rsi — 1 

A(si)A(s 2 ) ■ ■ ■ A(st) dsi... ds 2 ds 1 + ... (C.27) 
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This is obtained formally when solving by iteration for X(-) the fixed-point 
equation 

X(t) = 1+ / A{s)X{s)ds 

corresponding to the initial-value problem (C.24). The series (C.27) converges 
uniformly on compacts with respect to a, t. Indeed, since the entries of A(-) are 
locally bounded, there is for each finite interval X a constant K such that 

\A{s)\<K 

for almost all s£/; therefore the Ith term of the series is bounded by 

K l (t-a) 1 

The partial sums of the series obtained by differentiating each term with respect 
to t are simply the products of A(t) by the partial sums of the above series. 
Therefore that termwise differentiated series converges uniformly, too. It follows 
that the derivative of the sum exists and it equals this termwise derivative, that 
is, A(t)Q(t, a). So $(£,cr) is a solution for (C.24) in [a, t] and hence is the 
fundamental solution matrix introduced earlier. 

An advantage of this derivation is that it provides an explicit formula for 
the solution, in terms of the power series (C.27). This forms the basis of some 
numerical integration methods for linear systems. For constant matrices, it is 
the same as the series defining the exponential. 

Exercise C.4.2 Substitute the Peano-Baker series for $(£, er) into the variation 
of parameters formula (C.26), assuming for simplicity the case a; = 0, and 
obtain in this manner a series expansion for £(£). □ 

Remark C.4.3 When A(t) is analytic (respectively, /c-times continuously dif- 
ferentiable, smooth) on an open interval X (meaning that each entry of A is, as a 
function oft), then also $(£, s) is analytic (respectively, fc+1-times continuously 
differentiable, smooth), as a function of (£, s) G X x X. This is because, for any 
fixed cGI, 

$(t,s) = <^{t,a)^(s,(7y 1 , 

and each factor has the desired degree of smoothness, by Propositions C.3.11 
and C.3.12, being solutions of Equation (C.24). □ 

Exercise C.4.4 For discrete-time difference equations, one may define 

$(t,a) := A{t- l)A{t-2)...A{a) 

whenever t > a, and 

$(cr,cr) :=/. 
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Derive then an expansion analogous to that in equation (C.26) for the discrete- 
time case. Note that in the special situation in which every matrix A(k) is 
invertible, one can then also define 

*(*,*) :=*(M) _1 

for t < s, and Property (b) in Lemma C.4.1 holds. □ 

C.5 Stability of Linear Equations 

We review here some basic facts about the asymptotic behavior of linear differ- 
ential and difference equations. 

Proposition C.5.1 Let A be an n x n real (respectively complex) matrix. 
Then: 

1. All real (respectively complex) solutions of the differential equation 

i(t) = A£(t) 

satisfy that 

£(£) -> as t -> oo (C.28) 

if and only if all eigenvalues of A have negative real parts. 

2. All real (respectively, complex) solutions of the difference equation 

S(t + l) = A$(t) 

satisfy (C.28) if and only if all eigenvalues of A have magnitude less than 
1. 

Proof. From the Jordan form of A it follows that each entry of e is a finite 
linear combination of terms of the form 

cf e xt , 

where A is an eigenvalue of A, i is an integer, and c is a constant, and that each 
entry of A k is a finite linear combination of terms of the form 

ck l X k 

(see Lemma A. 3. 2 in Appendix A. 3). Thus, e —» or A k — > if the respective 
eigenvalue condition holds. 

Conversely, assume that there is some eigenvalue A of A with 

a := Re A > . 
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Let v be a corresponding eigenvector. Then 

\\e tA v\\ =e at \\v\\ . 

This is constant if a = 0, and tends to oo if a > 0. If K = C, this means that 
the solution e v contradicts the assumption on A that all solutions converge 
to zero. If K = R, then the real and imaginary parts of e tA v are solutions of 

which cannot both converge to zero, again providing a contradiction. If there is 
some eigenvalue A with |A| > 1, an analogous argument applies to L4 v II. I 

Definition C.5.2 The complex matrix A is said to be 

• Hurwitz if all its eigenvalues have a negative real part; 

• convergent or discrete-time Hurwitz if all of its eigenvalues have magni- 
tude less than 1. □ 

Some authors call a Schur matrix what we are calling here a convergent 
matrix. 

It is possible to determine whether a given matrix is Hurwitz (or convergent) 
without having to compute eigenvalues, directly, by checking certain polynomial 
inequalities on the coefficients of A. This is typically done through a Routh- 
Hurwitz type of test; see, for instance, [34] for extensive discussions of such 
issues, and also Remark 5.7.20 in Chapter 5. 
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In addition to the references cited in the text, listed here are some additional 
papers and books. These contain many further references, and a few provide 
introductions to some of the areas that have not been covered, but the list is in 
no way comprehensive or balanced in terms of the different possible topics. 

Other Books 

Some books that should be of interest are included in this Bibliography. The 
three-part book [230] was in a sense the first "modern systems and control 
theory" treatise. Algebraic topics on realization are covered there, and the close 
analogies between automata and linear systems results are discussed. Though 
somewhat out of date, it is still worth reading. 

There are many automata theory texts; some examples include: [52], [97] 
(material on observability), [125] (an excellent treatise of many topics in au- 
tomata theory, very algebraic in character; it also includes a chapter on linear 
systems as well as many results on what can be described as "systems over 
semirings," a class that includes both linear systems and automata), and [156]. 

For general topics on linear systems, good references are for instance [17], 
[58] (focuses on performance issues), [65] (a classic, but apparently out of print), 
[88], [114] (a brief introduction to basic facts about linear systems, including 
controllability, observability, and minimality), [144] (a very detailed reference 
for elementary algebraic theory of linear systems and classical control theory; it 
has many realistic examples), [212] (modern, very algebraic, treatment of linear 
systems, emphasizing the 'polynomial matrix' approach), [314] (an elementary 
introduction to control theory, including comparisons to automata and a brief 
treatment of some 'advanced' topics such as category-theoretic methods), [335] 
(a rather summarized treatment of algebraic theory of time-invariant linear 
systems), [337] (introduces linear systems and optimal linear control, including 
many stochastic topics), and [430] (a very algebraic approach to linear regulator 
design, based on the author's "geometric" method of solution). 

There are literally hundreds of textbooks dealing with frequency-response 
and basic linear theory from an engineering standpoint, geared to an under- 
graduate audience but very useful nonetheless for learning the basic concepts. 
One particularly nice book is [260], as it contains very detailed examples of 
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applications. 

In optimal control the choice is also very large. For excellent introductions 
to quadratic/linear problems, including some study of the effect of nonlinear- 
ities as well as numerical methods, see [16] and [263]. More generally, some 
suggested texts are: [134] (one of the best treatises of optimal control; results 
proved rigorously and in great generality), [188] (especially for the time-optimal 
problem for linear systems), and [266] (an excellent "classical" reference, some- 
what outdated; exposition of linear and nonlinear optimal control, and of basic 
properties of nonlinear systems; little algebraic theory of linear systems). Also, 
[108] provides an excellent brief exposition to filtering and stochastic control. 
Among other books in optimal control are: [25], [44], [48], [103], [151], [165], 
[189], [247], [248], [276], [289], [315], [379], and [442]. 

On the control of nonlinear systems there are several books available, includ- 
ing: [115] (operator approach), [199], [311] (state-space methods, differential- 
geometric in character), [414] (a very nice exposition of material on ODEs rel- 
evant to control theory, including stability, as well as the operator approach to 
input/output nonlinear systems), and [422] (operator approach). 

For infinite-dimensional systems, some books are [107], [133], [147], and [281]. 
Many others are cited in them. 

For applications of control theory see for instance the following books: [20] 
(economics), [45] (management), [146] (automobiles), [168] (biology), [317] (eco- 
nomics), [325] (oil recovery), [328] (physiology), [346] (economics), [347] (man- 
agement), [350] (economics and engineering applications), [380] (chemical pro- 
cess control), [394] (biomedicine), [401] (psychology), [420] (robotics), and [427] 
(signal processing). Also, the Systems & Control Encyclopedia [352] and the 
The Control Handbook [274] contain many articles on applications as well as on 
further theoretical questions. 
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equation, 301 
map, 31 
i/s behavior of system, 31 
ISS, 330, 345 

Jacobian, 462 
Jordan 

block, 452 

form, 452 

/C-function, 330 
K aim an 

controllability decomposition, 

93 
filtering, 18, 321, 376 

steady-state, 390 
rank condition 

for controllability, 89 
for observability, 271 
/Coo -function, 330 
KX-function, 330 
knife edge, 164 
Kronecker indices, 190 

Lagrangian subspace, 386 
Laplace transform, 332 
LaSalle invariance principle, 226 
Laurent 

expansion, 300 

series, 297 
least-squares, 450 

recursive, 395 
length of control, 27 
Lie 

algebra, 143 



generated, 143 
bracket, 142 
derivative, 142, 218 
linear 

behavior 

continuous-time, 48 
discrete-time, 37 
system, 11, 47, 82, 264 
continuous-time, 47 
discrete-time, 36 
finite-dimensional, 36, 47 
with output, 17 
linear growth on controls, 65 
linear-quadratic problem, 364, 369 
linearization, 207 
feedback, 198 

of continuous-time system, 51 
of discrete-time system, 39 
under feedback, 21 
linearization principle, 5 
for controllability, 125 
for observability, 280 
for stability, 234 
local in time, 45 
locally 

absolutely continuous, 471 
controllable, 124 
integrable, 470 
observable, 280 
LQG, 395 

Lyapunov function, 219, 224, 391 
converse theorems, 258 

machine, 26 
Markov 

parameters, 38, 284 

sequence, 284 
Massera's Theorem, 258, 260 
matrix 

identities over rings, 460 

norm, 447 
maximal solution, 477 
Maximum Principle, 418, 433 

time-optimal, 434 
McMillan degree, 299 
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Mean Value Theorem, 60 
measurable function, 468 
measurement, 16 

map, 27 

value, 27, 30 
memoryless i/o map, 78 
minimal 

realization, 286 

surface of revolution, 413 

system, 308 

triple, 286 
Minimum Principle, 418 
moduli problems, 296 
Moore-Penrose inverse, 450 
multidimensional systems, 77 
multipliers, 348 

natural frequency, 15 

Nerode equivalence relation, 309 

neural networks, 129, 139, 312 

controllability, 130 

observability, 310 
Newton's method, 416 
next-state map, 33 
no-drift systems, 158 
noise, effect of, 17 
nonhomogeneous linear systems, 96 
nonlinear gain, 331 
nontriviality axiom, 26 
normal problem in optimal control, 

437 
normed space, 463 
NP-hard, 181 
null-controllability, 84, 99 
numerical 

methods for optimal control, 
415 

pole-shifting, 188 
Nyquist criterion, 342 

observability, 20, 262, 263 
asymptotic, 317, 329 
decomposition, 272 
form, 292 
Gramian, 279 



rank condition, 282 

sampled, 275 
observable 

locally, 280 

pair, 272 

reduction, 305 

system, 263 
observables, 311 
observation space, 282 
observer, 4, 18, 318 

optimal, 378 

reduced order, 320 
operational amplifiers, 6 
operator norm, 447 
optimal 

control data, 350 

cost-to-go, 355 

observer, 378 

path, 355, 392 

trajectory, 355, 392 
OSS, 331 
output 

controllability, 112 

feedback, 17 

stabilization, 218 

value, 27, 30 
output-to-state stability, 331 

Pade approximation, 311 
pair 

equilibrium, 40, 51 

of matrices, 92 
parity-checker, 34, 309 
partial 

derivative, 465 

realization, 311 
partition of integer, 191 
path, 28 

path planning, 181 
PD, 4 

Peano-Baker formula, 489 
pencil of matrices, 190 
pendulum, inverted, 91, 188, 272, 

303, 326, 330 
performance index, 347 
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phase 

lag, 334 

lead, 334 

shift, 334 
PID, 19 
piecewise 

analytic feedback, 260 

constant control, 63 

continuous functions, 42 

linear feedback, 260 
pointwise in distribution, 169 
Pole-Shifting Theorem, 11, 14, 186 
polynomial system, 273, 274 
Pontryagin's Maximum (or 

Minimum) Principle, 348 
positive definite function, 219 
proper, 336 

function, 219 

group action, 295 
proportional feedback, 4 
proportional-derivative, 4 
proportional- integral-derivative, 19 
proximal subgradients, 394 
pseudoinverse, 107, 450 
pulse-width modulated control, 260 

quantization, 24 

radially unbounded, 220 
rank 

of Markov sequence, 287, 312 
Rank Theorem, 464 
rational, 298 
reachability, 81 

matrix, 92 
reachable 

set, 84 

space, 92 
readout map, 27 
realizable, 284 
realization, 20, 32, 284 

canonical, 284, 286 

in controllability form, 292 

in observability form, 292 

minimal, 286 



partial, 311 
recurrent nets, 129, 139 

controllability, 130 

observability, 310 
recursive Markov sequence, 290 
reduced order observers, 320 
reference signal, 15, 368, 371 
regular synthesis, 394 
relative degree, 205 
response map, 30 
restriction axiom, 26, 30 
reversible systems, 84 
rhs, 43 
Riccati Differential Equation, 364 

for filtering, 378 

indefinite, 395 
right-hand side, 43 
rigid body, momentum control, 

163, 241, 245 
robust control, 23 

sample-and-hold, 7 
sampled 

controllable, 100 

observable, 275 

system, 72 
sampling, 138 

times, 7 
Sampling Theorem, 310 
satellite, 163, 241, 245 
Schur matrix, 492 
selection theorems, 78 
semigroup axiom, 26 
semirings, 493 
sequential machines, 33 
set- valued map, 77 
Shannon's Theorem, 138, 310 
shift operator, 289 
shopping cart, 164 
similar systems, 183, 286 
similarity of realizations, 286 
singular 

systems, 77, 256 

value decomposition, 449 

values, 448 
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slice, 146 

sliding mode control, 260 

small-time local controllability, 

139, 432 
smooth 

continuous-time system, 45 

discrete-time system, 39 

mapping, 461 

solution of optimal control 
problem, 359 

stabilizability, 236 

vector field, 141 
smoothly varying system, 113 
solution of ode, 474 
space 

control-value, 26 

input-value, 26 

measurement-value, 27 

output-value, 27 
Spectral Mapping Theorem, 454 
stability, 258 

asymptotic, 211 

exponential, 259 

input-to-state, 330, 345 

internal, 328 

non-asymptotic, 237 

of time- varying systems, 232 

output-to-state, 331 
stabilizability, 236 
stabilizable, 214 
state, 12, 27 

equilibrium, 40, 51 

initial, 28, 31 

space, 11, 26 

terminal, 28 
state-output, 77 
static output feedback, 17 
steady-state 

error, 19 

Kalman filter, 390 

Kalman gain, 390 

output, 334 
steepest descent, 417 
STLC, 432 
stochastic 



filtering, 18, 376 

state estimation, 18, 376 

systems, 23 
strict causality, 31 
strictly proper, 336 
strongly locally controllable, 129 
structural controllability, 138 
submanifold, 146 
superposition principle, 264 
Sylvester equation, 230 
system, 26, 77 

affine, 96 

affine in controls, 57, 154, 246, 
281, 362, 392, 406 

analytically varying, 113 

asymptotically observable, 317 

autonomous, 29 

bilinear, 74 

canonical, 284, 305 

chaotic, 313 

classical dynamical, 29 

complete, 29 

continuous-time, 44 
analytic, 268 
smooth, 45 

delay-differential, 79 

descriptor, 77, 256 

detectable, 317 

differentiable, 39, 45 

discrete-event, 76 

discrete-time, 32 
differentiable, 39 
smooth, 39 

distributed, 79 

final-state observable, 266 

Hamiltonian, 313 

hereditary, 79 

hybrid, 24, 76 

infinite dimensional, 79, 494 

initialized, 31 

input-to-state stable, 330, 345 

internally stable, 328 

isomorphism, 306 

ISS, 330, 345 

large scale, 77 
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linear, 47, 82, 264, 493 
nonhomogeneous, 96 
minimal, 308 
morphism, 306 
multidimensional, 77 
observable, 263 
of class C k , 45 
on group, 313 
OSS, 331 

output-to-state stable, 331 
over ring, 79 
piece wise linear, 138 
polynomial, 273, 313 
reversible, 84 
sampled, 72 
similarity, 183, 286 
singular, 77, 256 
smoothly varying, 113 
state-output, 77 
time-invariant, 29 
time-reversed, 45 
time-varying, 30 
topological, 123 
underlying, 27 
unstable, 237 
with no controls, 29 
with outputs, 27, 45 
without drift, 158, 393 

tangent vector field, 146 
terminal state, 28 
time 

series, 77, 312 

set, 25 
time-invariant, 11 

i/o behavior, 32 

system, 29 
time-optimal control, 423 
time-reversed system, 45 
time- varying system, 30 

stability, 232 
topological system, 123 
tracking, 15, 371 

error, 372 
trajectory, 28 



transfer 

function, 333 

matrix, 333 
transition map, 26, 33 
triple 

canonical, 284 

controllable and observable, 
283 

minimal, 286 

UBIBO, 327 
uncontrollable 

modes, 94 

polynomial, 94 
unicycle, 164 
uniformly bounded-input 

bounded-output, 327 
uniqueness of canonical 

realizations, 286 
universal formula for feedback 

stabilization, 246 
unstable 

closed-loop poles, 342 

open-loop poles, 342 

system, 237 

value function, 355, 392, 443 
variation of parameters, 488 
vector field, 141 
verification principle, 356 
viscosity solution, 394, 396 
VLSI testing, 310 
Volterra 

expansion, 75 

series, 74 

weak 

controllability, 141 
convergence, 56, 424 
weakly coercive mapping, 220 
well-posed interconnection, 322 

Young tableaux, 193 



measure, 467 
of a system, 257 



